2024-01-13T09:34:20.693 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-13T09:34:20.694 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} --name yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515860 --verbose --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515860/orig.config.yaml 2024-01-13T09:34:20.730 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-13T09:34:20.814 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515860 branch: wip-yuri2-testing-2024-01-12-1128-pacific description: rados/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} email: yweinste@redhat.com first_in_suite: false job_id: '7515860' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-yuri2-testing-2024-01-12-1128-pacific ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 445562ab4bc3ddfb386936119050695810860bcb ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 445562ab4bc3ddfb386936119050695810860bcb selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri2-testing-2024-01-12-1128-pacific sha1: 445562ab4bc3ddfb386936119050695810860bcb owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - - host.b - client.1 seed: 5246 sha1: 445562ab4bc3ddfb386936119050695810860bcb sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri2-testing-2024-01-12-1128-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 445562ab4bc3ddfb386936119050695810860bcb targets: smithi084.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB3VZXVJ3zgggvP/1bs8JLYarhJK4t2BfPZfKm5zL56Nc6E0uXQRjD0vHFw+UIgyz6eYMiH8a50HG6MBLpL5Yr4= smithi102.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJK2s6PnhQ1UyCQIe+xYA2Q3/b8STcSd/k2D6xm/EUEbCCrF+MAbmPmHWTxTnuqDC11N05b67hwZNZjGxpTskn0= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:3.0 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - nvme_loop: null - cephadm: roleless: true - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - vip: null - cephadm.shell: host.a: - ceph orch device ls --refresh - vip.exec: all-hosts: - systemctl stop nfs-server - cephadm.shell: host.a: - ceph orch apply rgw foorgw --port 8800 - ceph nfs cluster create foo --ingress --virtual-ip {{VIP0}}/{{VIPPREFIXLEN}} - vip.exec: host.a: - dnf install -y python3-boto3 || apt install -y python3-boto3 - /home/ubuntu/cephtest/cephadm shell radosgw-admin user create --uid foouser --display-name foo > /tmp/user.json - python: host.a: "import boto3\nimport json\n\nwith open('/tmp/user.json', 'rt') as f:\n\ \ info = json.loads(f.read())\ns3 = boto3.resource(\n 's3',\n aws_access_key_id=info['keys'][0]['access_key'],\n\ \ aws_secret_access_key=info['keys'][0]['secret_key'],\n endpoint_url='http://localhost:8800',\n\ )\nbucket = s3.Bucket('foobucket')\nbucket.create()\nbucket.put_object(Key='myobject',\ \ Body='thebody')\n" - cephadm.shell: host.a: - ceph nfs export create rgw --cluster-id foo --pseudo-path /foouser --user-id foouser - cephadm.wait_for_service: service: nfs.foo - cephadm.wait_for_service: service: ingress.nfs.foo - vip.exec: host.a: - mkdir /mnt/foo - sleep 5 - mount -t nfs {{VIP0}}:/foouser /mnt/foo - test -d /mnt/foo/foobucket - find /mnt/foo -ls - grep thebody /mnt/foo/foobucket/myobject - echo test > /mnt/foo/foobucket/newobject - sync - python: host.a: "import boto3\nimport json\nfrom io import BytesIO\n\nwith open('/tmp/user.json',\ \ 'rt') as f:\n info = json.loads(f.read())\ns3 = boto3.resource(\n 's3',\n\ \ aws_access_key_id=info['keys'][0]['access_key'],\n aws_secret_access_key=info['keys'][0]['secret_key'],\n\ \ endpoint_url='http://localhost:8800',\n)\nbucket = s3.Bucket('foobucket')\n\ data = BytesIO()\nbucket.download_fileobj(Fileobj=data, Key='newobject')\nprint(data.getvalue())\n\ assert data.getvalue().decode() == 'test\\n'\n" - vip.exec: host.a: - umount /mnt/foo - cephadm.shell: host.a: - ceph nfs export rm foo /foouser - ceph nfs cluster rm foo - cephadm.shell: host.a: - stat -c '%u %g' /var/log/ceph | grep '167 167' - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch ls | grep '^osd.all-available-devices ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-12_21:52:25 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-01-13T09:34:20.815 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa; will attempt to use it 2024-01-13T09:34:20.815 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks 2024-01-13T09:34:20.816 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-13T09:34:20.817 INFO:teuthology.task.internal:Checking packages... 2024-01-13T09:34:20.844 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '445562ab4bc3ddfb386936119050695810860bcb' 2024-01-13T09:34:20.844 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-13T09:34:20.844 INFO:teuthology.packaging:ref: None 2024-01-13T09:34:20.844 INFO:teuthology.packaging:tag: None 2024-01-13T09:34:20.844 INFO:teuthology.packaging:branch: wip-yuri2-testing-2024-01-12-1128-pacific 2024-01-13T09:34:20.844 INFO:teuthology.packaging:sha1: 445562ab4bc3ddfb386936119050695810860bcb 2024-01-13T09:34:20.845 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri2-testing-2024-01-12-1128-pacific 2024-01-13T09:34:21.082 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-456.g445562ab 2024-01-13T09:34:21.083 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-13T09:34:21.093 INFO:teuthology.task.internal:no buildpackages task found 2024-01-13T09:34:21.093 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-13T09:34:21.132 INFO:teuthology.task.internal:Saving configuration 2024-01-13T09:34:21.146 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-13T09:34:21.152 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-13T09:34:21.177 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi084.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515860', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-13 09:29:25.372783', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB3VZXVJ3zgggvP/1bs8JLYarhJK4t2BfPZfKm5zL56Nc6E0uXQRjD0vHFw+UIgyz6eYMiH8a50HG6MBLpL5Yr4='} 2024-01-13T09:34:21.203 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi102.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515860', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-13 09:29:25.371384', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJK2s6PnhQ1UyCQIe+xYA2Q3/b8STcSd/k2D6xm/EUEbCCrF+MAbmPmHWTxTnuqDC11N05b67hwZNZjGxpTskn0='} 2024-01-13T09:34:21.203 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-13T09:34:21.209 INFO:teuthology.task.internal:roles: ubuntu@smithi084.front.sepia.ceph.com - ['host.a', 'client.0'] 2024-01-13T09:34:21.209 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'client.1'] 2024-01-13T09:34:21.209 INFO:teuthology.run_tasks:Running task console_log... 2024-01-13T09:34:21.305 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd453bd0f70>, signals=[15]) 2024-01-13T09:34:21.306 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-13T09:34:21.312 INFO:teuthology.task.internal:Opening connections... 2024-01-13T09:34:21.312 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi084.front.sepia.ceph.com 2024-01-13T09:34:21.314 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-13T09:34:21.392 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi102.front.sepia.ceph.com 2024-01-13T09:34:21.393 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-13T09:34:21.476 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-13T09:34:21.488 DEBUG:teuthology.orchestra.run.smithi084:> uname -m 2024-01-13T09:34:21.512 INFO:teuthology.orchestra.run.smithi084.stdout:x86_64 2024-01-13T09:34:21.512 DEBUG:teuthology.orchestra.run.smithi084:> cat /etc/os-release 2024-01-13T09:34:21.572 INFO:teuthology.orchestra.run.smithi084.stdout:NAME="Red Hat Enterprise Linux" 2024-01-13T09:34:21.573 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION="8.6 (Ootpa)" 2024-01-13T09:34:21.573 INFO:teuthology.orchestra.run.smithi084.stdout:ID="rhel" 2024-01-13T09:34:21.573 INFO:teuthology.orchestra.run.smithi084.stdout:ID_LIKE="fedora" 2024-01-13T09:34:21.573 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION_ID="8.6" 2024-01-13T09:34:21.573 INFO:teuthology.orchestra.run.smithi084.stdout:PLATFORM_ID="platform:el8" 2024-01-13T09:34:21.573 INFO:teuthology.orchestra.run.smithi084.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-01-13T09:34:21.573 INFO:teuthology.orchestra.run.smithi084.stdout:ANSI_COLOR="0;31" 2024-01-13T09:34:21.573 INFO:teuthology.orchestra.run.smithi084.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-01-13T09:34:21.573 INFO:teuthology.orchestra.run.smithi084.stdout:HOME_URL="https://www.redhat.com/" 2024-01-13T09:34:21.573 INFO:teuthology.orchestra.run.smithi084.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-01-13T09:34:21.573 INFO:teuthology.orchestra.run.smithi084.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-13T09:34:21.573 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:34:21.573 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-13T09:34:21.574 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-01-13T09:34:21.574 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-01-13T09:34:21.574 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-01-13T09:34:21.574 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server 2024-01-13T09:34:21.599 DEBUG:teuthology.orchestra.run.smithi102:> uname -m 2024-01-13T09:34:21.619 INFO:teuthology.orchestra.run.smithi102.stdout:x86_64 2024-01-13T09:34:21.619 DEBUG:teuthology.orchestra.run.smithi102:> cat /etc/os-release 2024-01-13T09:34:21.678 INFO:teuthology.orchestra.run.smithi102.stdout:NAME="Red Hat Enterprise Linux" 2024-01-13T09:34:21.678 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION="8.6 (Ootpa)" 2024-01-13T09:34:21.678 INFO:teuthology.orchestra.run.smithi102.stdout:ID="rhel" 2024-01-13T09:34:21.678 INFO:teuthology.orchestra.run.smithi102.stdout:ID_LIKE="fedora" 2024-01-13T09:34:21.678 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION_ID="8.6" 2024-01-13T09:34:21.678 INFO:teuthology.orchestra.run.smithi102.stdout:PLATFORM_ID="platform:el8" 2024-01-13T09:34:21.678 INFO:teuthology.orchestra.run.smithi102.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-01-13T09:34:21.678 INFO:teuthology.orchestra.run.smithi102.stdout:ANSI_COLOR="0;31" 2024-01-13T09:34:21.678 INFO:teuthology.orchestra.run.smithi102.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-01-13T09:34:21.679 INFO:teuthology.orchestra.run.smithi102.stdout:HOME_URL="https://www.redhat.com/" 2024-01-13T09:34:21.679 INFO:teuthology.orchestra.run.smithi102.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-01-13T09:34:21.679 INFO:teuthology.orchestra.run.smithi102.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-13T09:34:21.679 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-01-13T09:34:21.679 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-13T09:34:21.679 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-01-13T09:34:21.679 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-01-13T09:34:21.679 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-01-13T09:34:21.679 INFO:teuthology.lock.ops:Updating smithi102.front.sepia.ceph.com on lock server 2024-01-13T09:34:21.699 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-13T09:34:21.707 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-13T09:34:21.726 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-13T09:34:21.727 DEBUG:teuthology.orchestra.run.smithi084:> test '!' -e /home/ubuntu/cephtest 2024-01-13T09:34:21.729 DEBUG:teuthology.orchestra.run.smithi102:> test '!' -e /home/ubuntu/cephtest 2024-01-13T09:34:21.747 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-13T09:34:21.752 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-13T09:34:21.753 DEBUG:teuthology.orchestra.run.smithi084:> test -z $(ls -A /var/lib/ceph) 2024-01-13T09:34:21.789 DEBUG:teuthology.orchestra.run.smithi102:> test -z $(ls -A /var/lib/ceph) 2024-01-13T09:34:21.833 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-13T09:34:21.890 INFO:teuthology.run_tasks:Running task kernel... 2024-01-13T09:34:21.905 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-01-13T09:34:21.905 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-01-13T09:34:21.906 DEBUG:teuthology.orchestra.run.smithi084:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-13T09:34:21.906 DEBUG:teuthology.orchestra.run.smithi102:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-13T09:34:21.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T09:34:21.925 DEBUG:teuthology.orchestra.run.smithi102:> uname -r 2024-01-13T09:34:21.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T09:34:21.926 DEBUG:teuthology.orchestra.run.smithi084:> uname -r 2024-01-13T09:34:21.983 INFO:teuthology.orchestra.run.smithi102.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-01-13T09:34:21.983 INFO:teuthology.task.kernel:Running kernel on smithi102: 4.18.0-513.9.1.el8_9.x86_64 2024-01-13T09:34:21.983 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum install -y kernel 2024-01-13T09:34:21.986 INFO:teuthology.orchestra.run.smithi084.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-01-13T09:34:21.986 INFO:teuthology.task.kernel:Running kernel on smithi084: 4.18.0-513.9.1.el8_9.x86_64 2024-01-13T09:34:21.987 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum install -y kernel 2024-01-13T09:34:23.478 INFO:teuthology.orchestra.run.smithi084.stdout:Updating Subscription Management repositories. 2024-01-13T09:34:23.478 INFO:teuthology.orchestra.run.smithi084.stdout:Unable to read consumer identity 2024-01-13T09:34:23.534 INFO:teuthology.orchestra.run.smithi102.stdout:Updating Subscription Management repositories. 2024-01-13T09:34:23.534 INFO:teuthology.orchestra.run.smithi102.stdout:Unable to read consumer identity 2024-01-13T09:34:23.793 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:34:23.793 INFO:teuthology.orchestra.run.smithi084.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-13T09:34:23.794 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:34:23.831 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-01-13T09:34:23.831 INFO:teuthology.orchestra.run.smithi102.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-13T09:34:23.831 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-01-13T09:34:25.463 INFO:teuthology.orchestra.run.smithi102.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2024-01-13T09:34:25.595 INFO:teuthology.orchestra.run.smithi084.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2024-01-13T09:34:26.914 INFO:teuthology.orchestra.run.smithi084.stdout:Extra Packages for Enterprise Linux 12 MB/s | 16 MB 00:01 2024-01-13T09:34:27.199 INFO:teuthology.orchestra.run.smithi102.stdout:Extra Packages for Enterprise Linux 9.4 MB/s | 16 MB 00:01 2024-01-13T09:34:30.471 INFO:teuthology.orchestra.run.smithi084.stdout:lab-extras 391 kB/s | 24 kB 00:00 2024-01-13T09:34:30.847 INFO:teuthology.orchestra.run.smithi102.stdout:lab-extras 672 kB/s | 24 kB 00:00 2024-01-13T09:34:31.232 INFO:teuthology.orchestra.run.smithi084.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-01-13T09:34:31.232 INFO:teuthology.orchestra.run.smithi084.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-01-13T09:34:31.319 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-01-13T09:34:31.320 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-01-13T09:34:31.320 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-01-13T09:34:31.433 DEBUG:teuthology.orchestra.run.smithi084:> echo no | sudo yum reinstall kernel || true 2024-01-13T09:34:31.558 INFO:teuthology.orchestra.run.smithi102.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-01-13T09:34:31.558 INFO:teuthology.orchestra.run.smithi102.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-01-13T09:34:31.649 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-01-13T09:34:31.650 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-01-13T09:34:31.650 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-01-13T09:34:31.796 DEBUG:teuthology.orchestra.run.smithi102:> echo no | sudo yum reinstall kernel || true 2024-01-13T09:34:31.810 INFO:teuthology.orchestra.run.smithi084.stdout:Updating Subscription Management repositories. 2024-01-13T09:34:31.810 INFO:teuthology.orchestra.run.smithi084.stdout:Unable to read consumer identity 2024-01-13T09:34:32.071 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:34:32.071 INFO:teuthology.orchestra.run.smithi084.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-13T09:34:32.071 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:34:32.130 INFO:teuthology.orchestra.run.smithi102.stdout:Updating Subscription Management repositories. 2024-01-13T09:34:32.131 INFO:teuthology.orchestra.run.smithi102.stdout:Unable to read consumer identity 2024-01-13T09:34:32.347 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-01-13T09:34:32.347 INFO:teuthology.orchestra.run.smithi102.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-13T09:34:32.347 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-01-13T09:34:32.381 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:00:02 ago on Sat 13 Jan 2024 09:34:30 AM UTC. 2024-01-13T09:34:32.450 INFO:teuthology.orchestra.run.smithi084.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-01-13T09:34:32.457 INFO:teuthology.orchestra.run.smithi084.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-01-13T09:34:32.490 INFO:teuthology.orchestra.run.smithi084.stderr:Error: No packages marked for reinstall. 2024-01-13T09:34:32.537 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum reinstall -y kernel || true 2024-01-13T09:34:32.655 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:02 ago on Sat 13 Jan 2024 09:34:30 AM UTC. 2024-01-13T09:34:32.730 INFO:teuthology.orchestra.run.smithi102.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-01-13T09:34:32.730 INFO:teuthology.orchestra.run.smithi102.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-01-13T09:34:32.773 INFO:teuthology.orchestra.run.smithi102.stderr:Error: No packages marked for reinstall. 2024-01-13T09:34:32.821 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum reinstall -y kernel || true 2024-01-13T09:34:32.871 INFO:teuthology.orchestra.run.smithi084.stdout:Updating Subscription Management repositories. 2024-01-13T09:34:32.872 INFO:teuthology.orchestra.run.smithi084.stdout:Unable to read consumer identity 2024-01-13T09:34:33.042 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:34:33.042 INFO:teuthology.orchestra.run.smithi084.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-13T09:34:33.042 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:34:33.155 INFO:teuthology.orchestra.run.smithi102.stdout:Updating Subscription Management repositories. 2024-01-13T09:34:33.155 INFO:teuthology.orchestra.run.smithi102.stdout:Unable to read consumer identity 2024-01-13T09:34:33.342 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:00:03 ago on Sat 13 Jan 2024 09:34:30 AM UTC. 2024-01-13T09:34:33.370 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-01-13T09:34:33.371 INFO:teuthology.orchestra.run.smithi102.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-13T09:34:33.371 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-01-13T09:34:33.405 INFO:teuthology.orchestra.run.smithi084.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-01-13T09:34:33.405 INFO:teuthology.orchestra.run.smithi084.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-01-13T09:34:33.430 INFO:teuthology.orchestra.run.smithi084.stderr:Error: No packages marked for reinstall. 2024-01-13T09:34:33.476 DEBUG:teuthology.orchestra.run.smithi084:> rpm -q kernel | sort -rV | head -n 1 2024-01-13T09:34:33.640 INFO:teuthology.orchestra.run.smithi084.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-01-13T09:34:33.641 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-01-13T09:34:33.641 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-01-13T09:34:33.641 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-01-13T09:34:33.641 DEBUG:teuthology.orchestra.run.smithi084:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-13T09:34:33.667 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:03 ago on Sat 13 Jan 2024 09:34:30 AM UTC. 2024-01-13T09:34:33.683 INFO:teuthology.orchestra.run.smithi084.stdout:ttyS1 2024-01-13T09:34:33.696 DEBUG:teuthology.parallel:result is None 2024-01-13T09:34:33.730 INFO:teuthology.orchestra.run.smithi102.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-01-13T09:34:33.730 INFO:teuthology.orchestra.run.smithi102.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-01-13T09:34:33.763 INFO:teuthology.orchestra.run.smithi102.stderr:Error: No packages marked for reinstall. 2024-01-13T09:34:33.812 DEBUG:teuthology.orchestra.run.smithi102:> rpm -q kernel | sort -rV | head -n 1 2024-01-13T09:34:33.981 INFO:teuthology.orchestra.run.smithi102.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-01-13T09:34:33.981 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-01-13T09:34:33.981 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-01-13T09:34:33.981 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-01-13T09:34:33.981 DEBUG:teuthology.orchestra.run.smithi102:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-13T09:34:34.018 INFO:teuthology.orchestra.run.smithi102.stdout:ttyS1 2024-01-13T09:34:34.032 DEBUG:teuthology.parallel:result is None 2024-01-13T09:34:34.032 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-13T09:34:34.043 INFO:teuthology.task.internal:Creating test directory... 2024-01-13T09:34:34.043 DEBUG:teuthology.orchestra.run.smithi084:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-13T09:34:34.045 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-13T09:34:34.089 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-13T09:34:34.095 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-13T09:34:34.141 INFO:teuthology.task.internal:Creating archive directory... 2024-01-13T09:34:34.141 DEBUG:teuthology.orchestra.run.smithi084:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-13T09:34:34.143 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-13T09:34:34.172 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-13T09:34:34.179 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-13T09:34:34.180 DEBUG:teuthology.orchestra.run.smithi084:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-13T09:34:34.208 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-13T09:34:34.240 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-13T09:34:34.250 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-13T09:34:34.257 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-13T09:34:34.267 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-13T09:34:34.269 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-13T09:34:34.276 INFO:teuthology.task.internal:Configuring sudo... 2024-01-13T09:34:34.277 DEBUG:teuthology.orchestra.run.smithi084:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-13T09:34:34.279 DEBUG:teuthology.orchestra.run.smithi102:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-13T09:34:34.318 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-13T09:34:34.328 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-13T09:34:34.328 DEBUG:teuthology.orchestra.run.smithi084:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-13T09:34:34.346 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-13T09:34:34.375 DEBUG:teuthology.orchestra.run.smithi084:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-13T09:34:34.445 DEBUG:teuthology.orchestra.run.smithi084:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-13T09:34:34.498 DEBUG:teuthology.orchestra.run.smithi084:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-13T09:34:34.582 DEBUG:teuthology.orchestra.run.smithi084:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-13T09:34:34.625 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-01-13T09:34:34.625 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-13T09:34:34.691 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-13T09:34:34.734 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-13T09:34:34.785 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-13T09:34:34.873 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-13T09:34:34.911 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-01-13T09:34:34.911 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-13T09:34:34.978 DEBUG:teuthology.orchestra.run.smithi084:> sudo service rsyslog restart 2024-01-13T09:34:34.980 DEBUG:teuthology.orchestra.run.smithi102:> sudo service rsyslog restart 2024-01-13T09:34:35.024 INFO:teuthology.orchestra.run.smithi084.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-13T09:34:35.076 INFO:teuthology.orchestra.run.smithi102.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-13T09:34:35.498 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-13T09:34:35.506 INFO:teuthology.task.internal:Starting timer... 2024-01-13T09:34:35.507 INFO:teuthology.run_tasks:Running task pcp... 2024-01-13T09:34:35.545 INFO:teuthology.run_tasks:Running task selinux... 2024-01-13T09:34:35.560 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-01-13T09:34:35.560 DEBUG:teuthology.orchestra.run.smithi084:> sudo service auditd rotate 2024-01-13T09:34:35.620 INFO:teuthology.orchestra.run.smithi084.stdout:Rotating logs: 2024-01-13T09:34:35.622 DEBUG:teuthology.orchestra.run.smithi102:> sudo service auditd rotate 2024-01-13T09:34:35.691 INFO:teuthology.orchestra.run.smithi102.stdout:Rotating logs: 2024-01-13T09:34:35.693 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-13T09:34:35.693 DEBUG:teuthology.orchestra.run.smithi084:> /usr/sbin/getenforce 2024-01-13T09:34:35.720 INFO:teuthology.orchestra.run.smithi084.stdout:Permissive 2024-01-13T09:34:35.720 DEBUG:teuthology.orchestra.run.smithi102:> /usr/sbin/getenforce 2024-01-13T09:34:35.738 INFO:teuthology.orchestra.run.smithi102.stdout:Permissive 2024-01-13T09:34:35.739 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi084.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi102.front.sepia.ceph.com': 'permissive'} 2024-01-13T09:34:35.739 DEBUG:teuthology.orchestra.run.smithi084:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-13T09:34:35.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T09:34:35.788 DEBUG:teuthology.orchestra.run.smithi102:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-13T09:34:35.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T09:34:35.815 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-13T09:34:35.815 DEBUG:teuthology.orchestra.run.smithi084:> sudo /usr/sbin/setenforce permissive 2024-01-13T09:34:35.856 DEBUG:teuthology.orchestra.run.smithi102:> sudo /usr/sbin/setenforce permissive 2024-01-13T09:34:35.883 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-13T09:34:35.893 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-13T09:34:35.911 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-13T09:34:35.912 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi084.front.sepia.ceph.com,smithi102.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-13T09:42:41.160 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi084.front.sepia.ceph.com'), Remote(name='ubuntu@smithi102.front.sepia.ceph.com')] 2024-01-13T09:42:41.162 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-01-13T09:42:41.163 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-13T09:42:41.251 DEBUG:teuthology.orchestra.run.smithi084:> true 2024-01-13T09:42:41.319 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-01-13T09:42:41.319 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-01-13T09:42:41.320 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-13T09:42:41.399 DEBUG:teuthology.orchestra.run.smithi102:> true 2024-01-13T09:42:41.470 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-01-13T09:42:41.470 INFO:teuthology.run_tasks:Running task clock... 2024-01-13T09:42:41.481 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-13T09:42:41.481 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-13T09:42:41.482 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-13T09:42:41.484 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-13T09:42:41.484 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-13T09:42:41.514 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-13T09:42:41.529 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-13T09:42:41.544 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-13T09:42:41.557 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-13T09:42:41.592 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: ntpd: command not found 2024-01-13T09:42:41.604 INFO:teuthology.orchestra.run.smithi084.stdout:506 Cannot talk to daemon 2024-01-13T09:42:41.619 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-13T09:42:41.633 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-13T09:42:41.636 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: ntpd: command not found 2024-01-13T09:42:41.648 INFO:teuthology.orchestra.run.smithi102.stdout:506 Cannot talk to daemon 2024-01-13T09:42:41.663 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-13T09:42:41.677 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-13T09:42:41.707 INFO:teuthology.orchestra.run.smithi084.stderr:bash: ntpq: command not found 2024-01-13T09:42:41.710 INFO:teuthology.orchestra.run.smithi084.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-13T09:42:41.710 INFO:teuthology.orchestra.run.smithi084.stdout:=============================================================================== 2024-01-13T09:42:41.710 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T09:42:41.710 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T09:42:41.710 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T09:42:41.710 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T09:42:41.752 INFO:teuthology.orchestra.run.smithi102.stderr:bash: ntpq: command not found 2024-01-13T09:42:41.755 INFO:teuthology.orchestra.run.smithi102.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-13T09:42:41.755 INFO:teuthology.orchestra.run.smithi102.stdout:=============================================================================== 2024-01-13T09:42:41.755 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T09:42:41.755 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T09:42:41.755 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T09:42:41.755 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T09:42:41.756 INFO:teuthology.run_tasks:Running task pexec... 2024-01-13T09:42:41.767 INFO:teuthology.task.pexec:Executing custom commands... 2024-01-13T09:42:41.767 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi084.front.sepia.ceph.com 2024-01-13T09:42:41.767 DEBUG:teuthology.orchestra.run.smithi084:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-13T09:42:41.768 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi102.front.sepia.ceph.com 2024-01-13T09:42:41.768 DEBUG:teuthology.orchestra.run.smithi102:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-13T09:42:42.144 INFO:teuthology.orchestra.run.smithi084.stdout:Updating Subscription Management repositories. 2024-01-13T09:42:42.175 INFO:teuthology.orchestra.run.smithi102.stdout:Updating Subscription Management repositories. 2024-01-13T09:42:43.884 INFO:teuthology.orchestra.run.smithi084.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 46 kB/s | 4.5 kB 00:00 2024-01-13T09:42:44.046 INFO:teuthology.orchestra.run.smithi102.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 31 kB/s | 4.1 kB 00:00 2024-01-13T09:42:44.374 INFO:teuthology.orchestra.run.smithi102.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 41 kB/s | 4.5 kB 00:00 2024-01-13T09:42:44.527 INFO:teuthology.orchestra.run.smithi084.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 35 kB/s | 4.1 kB 00:00 2024-01-13T09:42:47.263 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-01-13T09:42:47.264 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-13T09:42:47.264 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-01-13T09:42:47.264 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-13T09:42:47.264 INFO:teuthology.orchestra.run.smithi084.stdout:Resetting modules: 2024-01-13T09:42:47.264 INFO:teuthology.orchestra.run.smithi084.stdout: container-tools 2024-01-13T09:42:47.264 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:42:47.264 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-01-13T09:42:47.264 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-13T09:42:47.264 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:42:47.398 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-01-13T09:42:47.399 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-01-13T09:42:47.399 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2024-01-13T09:42:47.399 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-01-13T09:42:47.399 INFO:teuthology.orchestra.run.smithi102.stdout:Resetting modules: 2024-01-13T09:42:47.399 INFO:teuthology.orchestra.run.smithi102.stdout: container-tools 2024-01-13T09:42:47.399 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-01-13T09:42:47.399 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-01-13T09:42:47.399 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-01-13T09:42:47.399 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-01-13T09:43:00.494 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-01-13T09:43:00.953 INFO:teuthology.orchestra.run.smithi084.stdout:Updating Subscription Management repositories. 2024-01-13T09:43:01.018 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-01-13T09:43:01.487 INFO:teuthology.orchestra.run.smithi102.stdout:Updating Subscription Management repositories. 2024-01-13T09:43:02.862 INFO:teuthology.orchestra.run.smithi084.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 58 kB/s | 4.5 kB 00:00 2024-01-13T09:43:03.415 INFO:teuthology.orchestra.run.smithi102.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 32 kB/s | 4.1 kB 00:00 2024-01-13T09:43:03.468 INFO:teuthology.orchestra.run.smithi084.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 54 kB/s | 4.1 kB 00:00 2024-01-13T09:43:03.723 INFO:teuthology.orchestra.run.smithi102.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 45 kB/s | 4.5 kB 00:00 2024-01-13T09:43:06.122 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-01-13T09:43:06.271 INFO:teuthology.orchestra.run.smithi084.stdout:========================================================================================================================== 2024-01-13T09:43:06.271 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-01-13T09:43:06.271 INFO:teuthology.orchestra.run.smithi084.stdout:========================================================================================================================== 2024-01-13T09:43:06.271 INFO:teuthology.orchestra.run.smithi084.stdout:Installing group/module packages: 2024-01-13T09:43:06.271 INFO:teuthology.orchestra.run.smithi084.stdout: buildah x86_64 1.19.9-6.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 6.6 M 2024-01-13T09:43:06.271 INFO:teuthology.orchestra.run.smithi084.stdout: cockpit-podman noarch 29-2.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 1.1 M 2024-01-13T09:43:06.271 INFO:teuthology.orchestra.run.smithi084.stdout: crun x86_64 0.18-3.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 185 k 2024-01-13T09:43:06.271 INFO:teuthology.orchestra.run.smithi084.stdout: skopeo x86_64 1:1.2.4-2.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 6.2 M 2024-01-13T09:43:06.271 INFO:teuthology.orchestra.run.smithi084.stdout: toolbox x86_64 0.0.99.3-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 2.2 M 2024-01-13T09:43:06.271 INFO:teuthology.orchestra.run.smithi084.stdout: udica noarch 0.2.4-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 51 k 2024-01-13T09:43:06.271 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-01-13T09:43:06.272 INFO:teuthology.orchestra.run.smithi084.stdout: yajl x86_64 2.1.0-12.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2024-01-13T09:43:06.272 INFO:teuthology.orchestra.run.smithi084.stdout:Downgrading: 2024-01-13T09:43:06.272 INFO:teuthology.orchestra.run.smithi084.stdout: conmon x86_64 2:2.0.26-3.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 51 k 2024-01-13T09:43:06.272 INFO:teuthology.orchestra.run.smithi084.stdout: container-selinux noarch 2:2.189.0-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 55 k 2024-01-13T09:43:06.272 INFO:teuthology.orchestra.run.smithi084.stdout: containernetworking-plugins x86_64 0.9.1-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 19 M 2024-01-13T09:43:06.272 INFO:teuthology.orchestra.run.smithi084.stdout: containers-common x86_64 1:1.2.4-2.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 91 k 2024-01-13T09:43:06.272 INFO:teuthology.orchestra.run.smithi084.stdout: criu x86_64 3.15-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 511 k 2024-01-13T09:43:06.272 INFO:teuthology.orchestra.run.smithi084.stdout: fuse-overlayfs x86_64 1.4.0-2.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 72 k 2024-01-13T09:43:06.272 INFO:teuthology.orchestra.run.smithi084.stdout: libslirp x86_64 4.3.1-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 69 k 2024-01-13T09:43:06.272 INFO:teuthology.orchestra.run.smithi084.stdout: podman x86_64 3.0.1-15.module+el8.7.0+18021+1e286cc8 rhel-8-for-x86_64-appstream-rpms 11 M 2024-01-13T09:43:06.272 INFO:teuthology.orchestra.run.smithi084.stdout: podman-catatonit x86_64 3.0.1-15.module+el8.7.0+18021+1e286cc8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-01-13T09:43:06.272 INFO:teuthology.orchestra.run.smithi084.stdout: podman-docker noarch 3.0.1-15.module+el8.7.0+18021+1e286cc8 rhel-8-for-x86_64-appstream-rpms 57 k 2024-01-13T09:43:06.272 INFO:teuthology.orchestra.run.smithi084.stdout: runc x86_64 1.0.0-73.rc95.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 2.9 M 2024-01-13T09:43:06.272 INFO:teuthology.orchestra.run.smithi084.stdout: slirp4netns x86_64 1.1.8-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 51 k 2024-01-13T09:43:06.272 INFO:teuthology.orchestra.run.smithi084.stdout:Installing module profiles: 2024-01-13T09:43:06.273 INFO:teuthology.orchestra.run.smithi084.stdout: container-tools/common 2024-01-13T09:43:06.273 INFO:teuthology.orchestra.run.smithi084.stdout:Enabling module streams: 2024-01-13T09:43:06.273 INFO:teuthology.orchestra.run.smithi084.stdout: container-tools 3.0 2024-01-13T09:43:06.273 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:43:06.273 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-01-13T09:43:06.273 INFO:teuthology.orchestra.run.smithi084.stdout:========================================================================================================================== 2024-01-13T09:43:06.273 INFO:teuthology.orchestra.run.smithi084.stdout:Install 7 Packages 2024-01-13T09:43:06.273 INFO:teuthology.orchestra.run.smithi084.stdout:Downgrade 12 Packages 2024-01-13T09:43:06.273 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:43:06.273 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 50 M 2024-01-13T09:43:06.274 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-01-13T09:43:06.468 INFO:teuthology.orchestra.run.smithi084.stdout:(1/19): conmon-2.0.26-3.module+el8.7.0+16533+44 266 kB/s | 51 kB 00:00 2024-01-13T09:43:06.526 INFO:teuthology.orchestra.run.smithi084.stdout:(2/19): criu-3.15-1.module+el8.7.0+16533+44634e 2.0 MB/s | 511 kB 00:00 2024-01-13T09:43:06.601 INFO:teuthology.orchestra.run.smithi084.stdout:(3/19): fuse-overlayfs-1.4.0-2.module+el8.7.0+1 539 kB/s | 72 kB 00:00 2024-01-13T09:43:06.683 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-01-13T09:43:06.685 INFO:teuthology.orchestra.run.smithi084.stdout:(4/19): libslirp-4.3.1-1.module+el8.7.0+16533+4 436 kB/s | 69 kB 00:00 2024-01-13T09:43:06.832 INFO:teuthology.orchestra.run.smithi102.stdout:========================================================================================================================== 2024-01-13T09:43:06.832 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-01-13T09:43:06.832 INFO:teuthology.orchestra.run.smithi102.stdout:========================================================================================================================== 2024-01-13T09:43:06.833 INFO:teuthology.orchestra.run.smithi102.stdout:Installing group/module packages: 2024-01-13T09:43:06.833 INFO:teuthology.orchestra.run.smithi102.stdout: buildah x86_64 1.19.9-6.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 6.6 M 2024-01-13T09:43:06.833 INFO:teuthology.orchestra.run.smithi102.stdout: cockpit-podman noarch 29-2.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 1.1 M 2024-01-13T09:43:06.833 INFO:teuthology.orchestra.run.smithi102.stdout: crun x86_64 0.18-3.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 185 k 2024-01-13T09:43:06.833 INFO:teuthology.orchestra.run.smithi102.stdout: skopeo x86_64 1:1.2.4-2.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 6.2 M 2024-01-13T09:43:06.833 INFO:teuthology.orchestra.run.smithi102.stdout: toolbox x86_64 0.0.99.3-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 2.2 M 2024-01-13T09:43:06.833 INFO:teuthology.orchestra.run.smithi102.stdout: udica noarch 0.2.4-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 51 k 2024-01-13T09:43:06.833 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-01-13T09:43:06.833 INFO:teuthology.orchestra.run.smithi102.stdout: yajl x86_64 2.1.0-12.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2024-01-13T09:43:06.833 INFO:teuthology.orchestra.run.smithi102.stdout:Downgrading: 2024-01-13T09:43:06.833 INFO:teuthology.orchestra.run.smithi102.stdout: conmon x86_64 2:2.0.26-3.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 51 k 2024-01-13T09:43:06.833 INFO:teuthology.orchestra.run.smithi102.stdout: container-selinux noarch 2:2.189.0-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 55 k 2024-01-13T09:43:06.833 INFO:teuthology.orchestra.run.smithi102.stdout: containernetworking-plugins x86_64 0.9.1-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 19 M 2024-01-13T09:43:06.833 INFO:teuthology.orchestra.run.smithi102.stdout: containers-common x86_64 1:1.2.4-2.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 91 k 2024-01-13T09:43:06.834 INFO:teuthology.orchestra.run.smithi102.stdout: criu x86_64 3.15-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 511 k 2024-01-13T09:43:06.834 INFO:teuthology.orchestra.run.smithi102.stdout: fuse-overlayfs x86_64 1.4.0-2.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 72 k 2024-01-13T09:43:06.834 INFO:teuthology.orchestra.run.smithi102.stdout: libslirp x86_64 4.3.1-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 69 k 2024-01-13T09:43:06.834 INFO:teuthology.orchestra.run.smithi102.stdout: podman x86_64 3.0.1-15.module+el8.7.0+18021+1e286cc8 rhel-8-for-x86_64-appstream-rpms 11 M 2024-01-13T09:43:06.834 INFO:teuthology.orchestra.run.smithi102.stdout: podman-catatonit x86_64 3.0.1-15.module+el8.7.0+18021+1e286cc8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-01-13T09:43:06.834 INFO:teuthology.orchestra.run.smithi102.stdout: podman-docker noarch 3.0.1-15.module+el8.7.0+18021+1e286cc8 rhel-8-for-x86_64-appstream-rpms 57 k 2024-01-13T09:43:06.834 INFO:teuthology.orchestra.run.smithi102.stdout: runc x86_64 1.0.0-73.rc95.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 2.9 M 2024-01-13T09:43:06.834 INFO:teuthology.orchestra.run.smithi102.stdout: slirp4netns x86_64 1.1.8-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 51 k 2024-01-13T09:43:06.834 INFO:teuthology.orchestra.run.smithi102.stdout:Installing module profiles: 2024-01-13T09:43:06.834 INFO:teuthology.orchestra.run.smithi102.stdout: container-tools/common 2024-01-13T09:43:06.834 INFO:teuthology.orchestra.run.smithi102.stdout:Enabling module streams: 2024-01-13T09:43:06.834 INFO:teuthology.orchestra.run.smithi102.stdout: container-tools 3.0 2024-01-13T09:43:06.834 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-01-13T09:43:06.834 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-01-13T09:43:06.834 INFO:teuthology.orchestra.run.smithi102.stdout:========================================================================================================================== 2024-01-13T09:43:06.834 INFO:teuthology.orchestra.run.smithi102.stdout:Install 7 Packages 2024-01-13T09:43:06.835 INFO:teuthology.orchestra.run.smithi102.stdout:Downgrade 12 Packages 2024-01-13T09:43:06.835 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-01-13T09:43:06.835 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 50 M 2024-01-13T09:43:06.835 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-01-13T09:43:06.910 INFO:teuthology.orchestra.run.smithi084.stdout:(5/19): containernetworking-plugins-0.9.1-1.mod 29 MB/s | 19 MB 00:00 2024-01-13T09:43:06.935 INFO:teuthology.orchestra.run.smithi084.stdout:(6/19): podman-docker-3.0.1-15.module+el8.7.0+1 229 kB/s | 57 kB 00:00 2024-01-13T09:43:06.998 INFO:teuthology.orchestra.run.smithi102.stdout:(1/19): criu-3.15-1.module+el8.7.0+16533+44634e 3.1 MB/s | 511 kB 00:00 2024-01-13T09:43:07.023 INFO:teuthology.orchestra.run.smithi102.stdout:(2/19): conmon-2.0.26-3.module+el8.7.0+16533+44 274 kB/s | 51 kB 00:00 2024-01-13T09:43:07.044 INFO:teuthology.orchestra.run.smithi084.stdout:(7/19): container-selinux-2.189.0-1.module+el8. 506 kB/s | 55 kB 00:00 2024-01-13T09:43:07.140 INFO:teuthology.orchestra.run.smithi102.stdout:(3/19): fuse-overlayfs-1.4.0-2.module+el8.7.0+1 507 kB/s | 72 kB 00:00 2024-01-13T09:43:07.157 INFO:teuthology.orchestra.run.smithi102.stdout:(4/19): libslirp-4.3.1-1.module+el8.7.0+16533+4 518 kB/s | 69 kB 00:00 2024-01-13T09:43:07.199 INFO:teuthology.orchestra.run.smithi084.stdout:(8/19): podman-3.0.1-15.module+el8.7.0+18021+1e 18 MB/s | 11 MB 00:00 2024-01-13T09:43:07.253 INFO:teuthology.orchestra.run.smithi084.stdout:(9/19): runc-1.0.0-73.rc95.module+el8.7.0+16533 8.4 MB/s | 2.9 MB 00:00 2024-01-13T09:43:07.269 INFO:teuthology.orchestra.run.smithi084.stdout:(10/19): containers-common-1.2.4-2.module+el8.7 404 kB/s | 91 kB 00:00 2024-01-13T09:43:07.395 INFO:teuthology.orchestra.run.smithi084.stdout:(11/19): cockpit-podman-29-2.module+el8.7.0+165 8.5 MB/s | 1.1 MB 00:00 2024-01-13T09:43:07.399 INFO:teuthology.orchestra.run.smithi102.stdout:(5/19): containernetworking-plugins-0.9.1-1.mod 33 MB/s | 19 MB 00:00 2024-01-13T09:43:07.424 INFO:teuthology.orchestra.run.smithi102.stdout:(6/19): podman-docker-3.0.1-15.module+el8.7.0+1 214 kB/s | 57 kB 00:00 2024-01-13T09:43:07.444 INFO:teuthology.orchestra.run.smithi084.stdout:(12/19): slirp4netns-1.1.8-1.module+el8.7.0+165 268 kB/s | 51 kB 00:00 2024-01-13T09:43:07.491 INFO:teuthology.orchestra.run.smithi102.stdout:(7/19): container-selinux-2.189.0-1.module+el8. 822 kB/s | 55 kB 00:00 2024-01-13T09:43:07.504 INFO:teuthology.orchestra.run.smithi084.stdout:(13/19): podman-catatonit-3.0.1-15.module+el8.7 1.0 MB/s | 323 kB 00:00 2024-01-13T09:43:07.529 INFO:teuthology.orchestra.run.smithi084.stdout:(14/19): crun-0.18-3.module+el8.7.0+16533+44634 1.3 MB/s | 185 kB 00:00 2024-01-13T09:43:07.584 INFO:teuthology.orchestra.run.smithi102.stdout:(8/19): runc-1.0.0-73.rc95.module+el8.7.0+16533 16 MB/s | 2.9 MB 00:00 2024-01-13T09:43:07.609 INFO:teuthology.orchestra.run.smithi102.stdout:(9/19): containers-common-1.2.4-2.module+el8.7. 778 kB/s | 91 kB 00:00 2024-01-13T09:43:07.646 INFO:teuthology.orchestra.run.smithi084.stdout:(15/19): udica-0.2.4-1.module+el8.7.0+16533+446 433 kB/s | 51 kB 00:00 2024-01-13T09:43:07.713 INFO:teuthology.orchestra.run.smithi084.stdout:(16/19): toolbox-0.0.99.3-1.module+el8.7.0+1653 11 MB/s | 2.2 MB 00:00 2024-01-13T09:43:07.767 INFO:teuthology.orchestra.run.smithi102.stdout:(10/19): podman-3.0.1-15.module+el8.7.0+18021+1 17 MB/s | 11 MB 00:00 2024-01-13T09:43:07.792 INFO:teuthology.orchestra.run.smithi102.stdout:(11/19): slirp4netns-1.1.8-1.module+el8.7.0+165 279 kB/s | 51 kB 00:00 2024-01-13T09:43:07.797 INFO:teuthology.orchestra.run.smithi084.stdout:(17/19): skopeo-1.2.4-2.module+el8.7.0+16533+44 18 MB/s | 6.2 MB 00:00 2024-01-13T09:43:07.818 INFO:teuthology.orchestra.run.smithi102.stdout:(12/19): podman-catatonit-3.0.1-15.module+el8.7 1.3 MB/s | 323 kB 00:00 2024-01-13T09:43:07.863 INFO:teuthology.orchestra.run.smithi084.stdout:(18/19): yajl-2.1.0-12.el8.x86_64.rpm 273 kB/s | 41 kB 00:00 2024-01-13T09:43:07.935 INFO:teuthology.orchestra.run.smithi102.stdout:(13/19): cockpit-podman-29-2.module+el8.7.0+165 6.4 MB/s | 1.1 MB 00:00 2024-01-13T09:43:07.977 INFO:teuthology.orchestra.run.smithi102.stdout:(14/19): crun-0.18-3.module+el8.7.0+16533+44634 1.0 MB/s | 185 kB 00:00 2024-01-13T09:43:07.980 INFO:teuthology.orchestra.run.smithi084.stdout:(19/19): buildah-1.19.9-6.module+el8.7.0+16533+ 20 MB/s | 6.6 MB 00:00 2024-01-13T09:43:07.981 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-01-13T09:43:07.981 INFO:teuthology.orchestra.run.smithi084.stdout:Total 29 MB/s | 50 MB 00:01 2024-01-13T09:43:08.102 INFO:teuthology.orchestra.run.smithi102.stdout:(15/19): toolbox-0.0.99.3-1.module+el8.7.0+1653 13 MB/s | 2.2 MB 00:00 2024-01-13T09:43:08.127 INFO:teuthology.orchestra.run.smithi102.stdout:(16/19): udica-0.2.4-1.module+el8.7.0+16533+446 336 kB/s | 51 kB 00:00 2024-01-13T09:43:08.219 INFO:teuthology.orchestra.run.smithi102.stdout:(17/19): skopeo-1.2.4-2.module+el8.7.0+16533+44 15 MB/s | 6.2 MB 00:00 2024-01-13T09:43:08.253 INFO:teuthology.orchestra.run.smithi102.stdout:(18/19): yajl-2.1.0-12.el8.x86_64.rpm 327 kB/s | 41 kB 00:00 2024-01-13T09:43:08.361 INFO:teuthology.orchestra.run.smithi102.stdout:(19/19): buildah-1.19.9-6.module+el8.7.0+16533+ 26 MB/s | 6.6 MB 00:00 2024-01-13T09:43:08.362 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-01-13T09:43:08.362 INFO:teuthology.orchestra.run.smithi102.stdout:Total 32 MB/s | 50 MB 00:01 2024-01-13T09:43:08.461 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-01-13T09:43:08.504 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-01-13T09:43:08.504 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-01-13T09:43:08.845 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-01-13T09:43:08.889 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-01-13T09:43:08.889 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-01-13T09:43:08.931 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-01-13T09:43:08.940 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-01-13T09:43:09.312 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-01-13T09:43:09.320 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-01-13T09:43:10.150 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-01-13T09:43:10.166 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: container-selinux-2:2.189.0-1.module+el8.7.0+16533+4 1/1 2024-01-13T09:43:10.198 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: container-selinux-2:2.189.0-1.module+el8.7.0+16533 1/31 2024-01-13T09:43:10.400 INFO:teuthology.orchestra.run.smithi084.stdout: Downgrading : container-selinux-2:2.189.0-1.module+el8.7.0+16533 1/31 2024-01-13T09:43:10.461 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-01-13T09:43:10.477 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: container-selinux-2:2.189.0-1.module+el8.7.0+16533+4 1/1 2024-01-13T09:43:10.512 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: container-selinux-2:2.189.0-1.module+el8.7.0+16533 1/31 2024-01-13T09:43:10.666 INFO:teuthology.orchestra.run.smithi102.stdout: Downgrading : container-selinux-2:2.189.0-1.module+el8.7.0+16533 1/31 2024-01-13T09:43:27.153 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: container-selinux-2:2.189.0-1.module+el8.7.0+16533 1/31 2024-01-13T09:43:27.362 INFO:teuthology.orchestra.run.smithi084.stdout: Downgrading : fuse-overlayfs-1.4.0-2.module+el8.7.0+16533+44634e 2/31 2024-01-13T09:43:27.380 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: fuse-overlayfs-1.4.0-2.module+el8.7.0+16533+44634e 2/31 2024-01-13T09:43:27.387 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: container-selinux-2:2.189.0-1.module+el8.7.0+16533 1/31 2024-01-13T09:43:27.572 INFO:teuthology.orchestra.run.smithi102.stdout: Downgrading : fuse-overlayfs-1.4.0-2.module+el8.7.0+16533+44634e 2/31 2024-01-13T09:43:27.590 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: fuse-overlayfs-1.4.0-2.module+el8.7.0+16533+44634e 2/31 2024-01-13T09:43:27.595 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : yajl-2.1.0-12.el8.x86_64 3/31 2024-01-13T09:43:27.788 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : yajl-2.1.0-12.el8.x86_64 3/31 2024-01-13T09:43:27.801 INFO:teuthology.orchestra.run.smithi084.stdout: Downgrading : libslirp-4.3.1-1.module+el8.7.0+16533+44634e24.x86 4/31 2024-01-13T09:43:27.970 INFO:teuthology.orchestra.run.smithi102.stdout: Downgrading : libslirp-4.3.1-1.module+el8.7.0+16533+44634e24.x86 4/31 2024-01-13T09:43:28.054 INFO:teuthology.orchestra.run.smithi084.stdout: Downgrading : slirp4netns-1.1.8-1.module+el8.7.0+16533+44634e24. 5/31 2024-01-13T09:43:28.054 INFO:teuthology.orchestra.run.smithi084.stdout: Downgrading : containers-common-1:1.2.4-2.module+el8.7.0+16533+4 6/31 2024-01-13T09:43:28.055 INFO:teuthology.orchestra.run.smithi084.stdout:warning: /etc/containers/registries.conf created as /etc/containers/registries.conf.rpmnew 2024-01-13T09:43:28.055 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:43:28.173 INFO:teuthology.orchestra.run.smithi102.stdout: Downgrading : slirp4netns-1.1.8-1.module+el8.7.0+16533+44634e24. 5/31 2024-01-13T09:43:28.173 INFO:teuthology.orchestra.run.smithi102.stdout: Downgrading : containers-common-1:1.2.4-2.module+el8.7.0+16533+4 6/31 2024-01-13T09:43:28.173 INFO:teuthology.orchestra.run.smithi102.stdout:warning: /etc/containers/registries.conf created as /etc/containers/registries.conf.rpmnew 2024-01-13T09:43:28.173 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-01-13T09:43:28.921 INFO:teuthology.orchestra.run.smithi084.stdout: Downgrading : criu-3.15-1.module+el8.7.0+16533+44634e24.x86_64 7/31 2024-01-13T09:43:28.990 INFO:teuthology.orchestra.run.smithi102.stdout: Downgrading : criu-3.15-1.module+el8.7.0+16533+44634e24.x86_64 7/31 2024-01-13T09:43:30.693 INFO:teuthology.orchestra.run.smithi084.stdout: Downgrading : runc-1.0.0-73.rc95.module+el8.7.0+16533+44634e24.x 8/31 2024-01-13T09:43:30.741 INFO:teuthology.orchestra.run.smithi102.stdout: Downgrading : runc-1.0.0-73.rc95.module+el8.7.0+16533+44634e24.x 8/31 2024-01-13T09:43:30.953 INFO:teuthology.orchestra.run.smithi102.stdout: Downgrading : containernetworking-plugins-0.9.1-1.module+el8.7.0 9/31 2024-01-13T09:43:30.968 INFO:teuthology.orchestra.run.smithi084.stdout: Downgrading : containernetworking-plugins-0.9.1-1.module+el8.7.0 9/31 2024-01-13T09:43:31.189 INFO:teuthology.orchestra.run.smithi102.stdout: Downgrading : conmon-2:2.0.26-3.module+el8.7.0+16533+44634e24.x8 10/31 2024-01-13T09:43:31.229 INFO:teuthology.orchestra.run.smithi084.stdout: Downgrading : conmon-2:2.0.26-3.module+el8.7.0+16533+44634e24.x8 10/31 2024-01-13T09:43:32.515 INFO:teuthology.orchestra.run.smithi102.stdout: Downgrading : podman-catatonit-3.0.1-15.module+el8.7.0+18021+1e2 11/31 2024-01-13T09:43:32.615 INFO:teuthology.orchestra.run.smithi084.stdout: Downgrading : podman-catatonit-3.0.1-15.module+el8.7.0+18021+1e2 11/31 2024-01-13T09:43:32.819 INFO:teuthology.orchestra.run.smithi102.stdout: Downgrading : podman-3.0.1-15.module+el8.7.0+18021+1e286cc8.x86_ 12/31 2024-01-13T09:43:32.930 INFO:teuthology.orchestra.run.smithi084.stdout: Downgrading : podman-3.0.1-15.module+el8.7.0+18021+1e286cc8.x86_ 12/31 2024-01-13T09:43:33.072 INFO:teuthology.orchestra.run.smithi102.stdout: Downgrading : podman-docker-3.0.1-15.module+el8.7.0+18021+1e286c 13/31 2024-01-13T09:43:33.270 INFO:teuthology.orchestra.run.smithi084.stdout: Downgrading : podman-docker-3.0.1-15.module+el8.7.0+18021+1e286c 13/31 2024-01-13T09:43:33.444 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : cockpit-podman-29-2.module+el8.7.0+16533+44634e24. 14/31 2024-01-13T09:43:33.668 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : cockpit-podman-29-2.module+el8.7.0+16533+44634e24. 14/31 2024-01-13T09:43:34.151 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : toolbox-0.0.99.3-1.module+el8.7.0+16533+44634e24.x 15/31 2024-01-13T09:43:34.434 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : toolbox-0.0.99.3-1.module+el8.7.0+16533+44634e24.x 15/31 2024-01-13T09:43:34.852 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : buildah-1.19.9-6.module+el8.7.0+16533+44634e24.x86 16/31 2024-01-13T09:43:35.070 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : skopeo-1:1.2.4-2.module+el8.7.0+16533+44634e24.x86 17/31 2024-01-13T09:43:35.155 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : buildah-1.19.9-6.module+el8.7.0+16533+44634e24.x86 16/31 2024-01-13T09:43:35.279 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : crun-0.18-3.module+el8.7.0+16533+44634e24.x86_64 18/31 2024-01-13T09:43:35.385 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : skopeo-1:1.2.4-2.module+el8.7.0+16533+44634e24.x86 17/31 2024-01-13T09:43:35.478 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : udica-0.2.4-1.module+el8.7.0+16533+44634e24.noarch 19/31 2024-01-13T09:43:35.597 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : podman-docker-3:4.6.1-4.module+el8.9.0+20326+38708 20/31 2024-01-13T09:43:35.602 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : crun-0.18-3.module+el8.7.0+16533+44634e24.x86_64 18/31 2024-01-13T09:43:35.611 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: podman-3:4.6.1-4.module+el8.9.0+20326+387084d0.x86 21/31 2024-01-13T09:43:35.611 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : podman-3:4.6.1-4.module+el8.9.0+20326+387084d0.x86 21/31 2024-01-13T09:43:35.834 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : udica-0.2.4-1.module+el8.7.0+16533+44634e24.noarch 19/31 2024-01-13T09:43:35.926 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: podman-3:4.6.1-4.module+el8.9.0+20326+387084d0.x86 21/31 2024-01-13T09:43:35.987 INFO:teuthology.orchestra.run.smithi084.stdout: Cleanup : podman-docker-3:4.6.1-4.module+el8.9.0+20326+38708 20/31 2024-01-13T09:43:36.000 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: podman-3:4.6.1-4.module+el8.9.0+20326+387084d0.x86 21/31 2024-01-13T09:43:36.001 INFO:teuthology.orchestra.run.smithi084.stdout: Cleanup : podman-3:4.6.1-4.module+el8.9.0+20326+387084d0.x86 21/31 2024-01-13T09:43:36.092 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : containers-common-2:1-70.module+el8.9.0+20326+3870 22/31 2024-01-13T09:43:36.092 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : container-selinux-2:2.221.0-1.module+el8.9.0+20326 23/31 2024-01-13T09:43:36.307 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: container-selinux-2:2.221.0-1.module+el8.9.0+20326 23/31 2024-01-13T09:43:36.307 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : podman-catatonit-3:4.6.1-4.module+el8.9.0+20326+38 24/31 2024-01-13T09:43:36.344 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: podman-3:4.6.1-4.module+el8.9.0+20326+387084d0.x86 21/31 2024-01-13T09:43:36.431 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: podman-catatonit-3:4.6.1-4.module+el8.9.0+20326+38 24/31 2024-01-13T09:43:36.568 INFO:teuthology.orchestra.run.smithi084.stdout: Cleanup : containers-common-2:1-70.module+el8.9.0+20326+3870 22/31 2024-01-13T09:43:36.569 INFO:teuthology.orchestra.run.smithi084.stdout: Cleanup : container-selinux-2:2.221.0-1.module+el8.9.0+20326 23/31 2024-01-13T09:43:36.572 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : slirp4netns-1.2.1-1.module+el8.9.0+20326+387084d0. 25/31 2024-01-13T09:43:36.722 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : runc-1:1.1.9-1.module+el8.9.0+20326+387084d0.x86_6 26/31 2024-01-13T09:43:36.796 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: container-selinux-2:2.221.0-1.module+el8.9.0+20326 23/31 2024-01-13T09:43:36.796 INFO:teuthology.orchestra.run.smithi084.stdout: Cleanup : podman-catatonit-3:4.6.1-4.module+el8.9.0+20326+38 24/31 2024-01-13T09:43:36.908 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : criu-3.18-4.module+el8.9.0+20326+387084d0.x86_64 27/31 2024-01-13T09:43:36.945 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: podman-catatonit-3:4.6.1-4.module+el8.9.0+20326+38 24/31 2024-01-13T09:43:37.055 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : libslirp-4.4.0-1.module+el8.9.0+20326+387084d0.x86 28/31 2024-01-13T09:43:37.128 INFO:teuthology.orchestra.run.smithi084.stdout: Cleanup : slirp4netns-1.2.1-1.module+el8.9.0+20326+387084d0. 25/31 2024-01-13T09:43:37.205 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : fuse-overlayfs-1.12-1.module+el8.9.0+20326+387084d 29/31 2024-01-13T09:43:37.303 INFO:teuthology.orchestra.run.smithi084.stdout: Cleanup : runc-1:1.1.9-1.module+el8.9.0+20326+387084d0.x86_6 26/31 2024-01-13T09:43:37.350 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : conmon-3:2.1.8-1.module+el8.9.0+20326+387084d0.x86 30/31 2024-01-13T09:43:37.350 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : containernetworking-plugins-1:1.3.0-4.module+el8.9 31/31 2024-01-13T09:43:37.510 INFO:teuthology.orchestra.run.smithi084.stdout: Cleanup : criu-3.18-4.module+el8.9.0+20326+387084d0.x86_64 27/31 2024-01-13T09:43:37.555 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: containernetworking-plugins-1:1.3.0-4.module+el8.9 31/31 2024-01-13T09:43:37.669 INFO:teuthology.orchestra.run.smithi084.stdout: Cleanup : libslirp-4.4.0-1.module+el8.9.0+20326+387084d0.x86 28/31 2024-01-13T09:43:37.886 INFO:teuthology.orchestra.run.smithi084.stdout: Cleanup : fuse-overlayfs-1.12-1.module+el8.9.0+20326+387084d 29/31 2024-01-13T09:43:38.081 INFO:teuthology.orchestra.run.smithi084.stdout: Cleanup : conmon-3:2.1.8-1.module+el8.9.0+20326+387084d0.x86 30/31 2024-01-13T09:43:38.081 INFO:teuthology.orchestra.run.smithi084.stdout: Cleanup : containernetworking-plugins-1:1.3.0-4.module+el8.9 31/31 2024-01-13T09:43:38.311 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: containernetworking-plugins-1:1.3.0-4.module+el8.9 31/31 2024-01-13T09:43:55.171 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: container-selinux-2:2.189.0-1.module+el8.7.0+16533 31/31 2024-01-13T09:43:55.648 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: containernetworking-plugins-1:1.3.0-4.module+el8.9 31/31 2024-01-13T09:43:55.648 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : conmon-2:2.0.26-3.module+el8.7.0+16533+44634e24.x8 1/31 2024-01-13T09:43:55.648 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : conmon-3:2.1.8-1.module+el8.9.0+20326+387084d0.x86 2/31 2024-01-13T09:43:55.648 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : containernetworking-plugins-0.9.1-1.module+el8.7.0 3/31 2024-01-13T09:43:55.648 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : containernetworking-plugins-1:1.3.0-4.module+el8.9 4/31 2024-01-13T09:43:55.648 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : criu-3.15-1.module+el8.7.0+16533+44634e24.x86_64 5/31 2024-01-13T09:43:55.648 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : criu-3.18-4.module+el8.9.0+20326+387084d0.x86_64 6/31 2024-01-13T09:43:55.648 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : fuse-overlayfs-1.4.0-2.module+el8.7.0+16533+44634e 7/31 2024-01-13T09:43:55.649 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : fuse-overlayfs-1.12-1.module+el8.9.0+20326+387084d 8/31 2024-01-13T09:43:55.649 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libslirp-4.3.1-1.module+el8.7.0+16533+44634e24.x86 9/31 2024-01-13T09:43:55.649 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libslirp-4.4.0-1.module+el8.9.0+20326+387084d0.x86 10/31 2024-01-13T09:43:55.649 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : podman-3.0.1-15.module+el8.7.0+18021+1e286cc8.x86_ 11/31 2024-01-13T09:43:55.649 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : podman-3:4.6.1-4.module+el8.9.0+20326+387084d0.x86 12/31 2024-01-13T09:43:55.649 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : podman-docker-3.0.1-15.module+el8.7.0+18021+1e286c 13/31 2024-01-13T09:43:55.649 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : podman-docker-3:4.6.1-4.module+el8.9.0+20326+38708 14/31 2024-01-13T09:43:55.650 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : runc-1.0.0-73.rc95.module+el8.7.0+16533+44634e24.x 15/31 2024-01-13T09:43:55.650 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : runc-1:1.1.9-1.module+el8.9.0+20326+387084d0.x86_6 16/31 2024-01-13T09:43:55.650 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : container-selinux-2:2.189.0-1.module+el8.7.0+16533 17/31 2024-01-13T09:43:55.650 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : container-selinux-2:2.221.0-1.module+el8.9.0+20326 18/31 2024-01-13T09:43:55.650 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : containers-common-1:1.2.4-2.module+el8.7.0+16533+4 19/31 2024-01-13T09:43:55.650 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : containers-common-2:1-70.module+el8.9.0+20326+3870 20/31 2024-01-13T09:43:55.651 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : podman-catatonit-3.0.1-15.module+el8.7.0+18021+1e2 21/31 2024-01-13T09:43:55.651 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : podman-catatonit-3:4.6.1-4.module+el8.9.0+20326+38 22/31 2024-01-13T09:43:55.651 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : slirp4netns-1.1.8-1.module+el8.7.0+16533+44634e24. 23/31 2024-01-13T09:43:55.651 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : slirp4netns-1.2.1-1.module+el8.9.0+20326+387084d0. 24/31 2024-01-13T09:43:55.651 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : cockpit-podman-29-2.module+el8.7.0+16533+44634e24. 25/31 2024-01-13T09:43:55.651 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : crun-0.18-3.module+el8.7.0+16533+44634e24.x86_64 26/31 2024-01-13T09:43:55.651 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : skopeo-1:1.2.4-2.module+el8.7.0+16533+44634e24.x86 27/31 2024-01-13T09:43:55.652 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : toolbox-0.0.99.3-1.module+el8.7.0+16533+44634e24.x 28/31 2024-01-13T09:43:55.652 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : udica-0.2.4-1.module+el8.7.0+16533+44634e24.noarch 29/31 2024-01-13T09:43:55.652 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : buildah-1.19.9-6.module+el8.7.0+16533+44634e24.x86 30/31 2024-01-13T09:43:55.934 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: container-selinux-2:2.189.0-1.module+el8.7.0+16533 31/31 2024-01-13T09:43:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: containernetworking-plugins-1:1.3.0-4.module+el8.9 31/31 2024-01-13T09:43:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : conmon-2:2.0.26-3.module+el8.7.0+16533+44634e24.x8 1/31 2024-01-13T09:43:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : conmon-3:2.1.8-1.module+el8.9.0+20326+387084d0.x86 2/31 2024-01-13T09:43:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : containernetworking-plugins-0.9.1-1.module+el8.7.0 3/31 2024-01-13T09:43:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : containernetworking-plugins-1:1.3.0-4.module+el8.9 4/31 2024-01-13T09:43:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : criu-3.15-1.module+el8.7.0+16533+44634e24.x86_64 5/31 2024-01-13T09:43:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : criu-3.18-4.module+el8.9.0+20326+387084d0.x86_64 6/31 2024-01-13T09:43:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : fuse-overlayfs-1.4.0-2.module+el8.7.0+16533+44634e 7/31 2024-01-13T09:43:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : fuse-overlayfs-1.12-1.module+el8.9.0+20326+387084d 8/31 2024-01-13T09:43:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libslirp-4.3.1-1.module+el8.7.0+16533+44634e24.x86 9/31 2024-01-13T09:43:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libslirp-4.4.0-1.module+el8.9.0+20326+387084d0.x86 10/31 2024-01-13T09:43:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : podman-3.0.1-15.module+el8.7.0+18021+1e286cc8.x86_ 11/31 2024-01-13T09:43:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : podman-3:4.6.1-4.module+el8.9.0+20326+387084d0.x86 12/31 2024-01-13T09:43:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : podman-docker-3.0.1-15.module+el8.7.0+18021+1e286c 13/31 2024-01-13T09:43:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : podman-docker-3:4.6.1-4.module+el8.9.0+20326+38708 14/31 2024-01-13T09:43:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : runc-1.0.0-73.rc95.module+el8.7.0+16533+44634e24.x 15/31 2024-01-13T09:43:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : runc-1:1.1.9-1.module+el8.9.0+20326+387084d0.x86_6 16/31 2024-01-13T09:43:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : container-selinux-2:2.189.0-1.module+el8.7.0+16533 17/31 2024-01-13T09:43:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : container-selinux-2:2.221.0-1.module+el8.9.0+20326 18/31 2024-01-13T09:43:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : containers-common-1:1.2.4-2.module+el8.7.0+16533+4 19/31 2024-01-13T09:43:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : containers-common-2:1-70.module+el8.9.0+20326+3870 20/31 2024-01-13T09:43:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : podman-catatonit-3.0.1-15.module+el8.7.0+18021+1e2 21/31 2024-01-13T09:43:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : podman-catatonit-3:4.6.1-4.module+el8.9.0+20326+38 22/31 2024-01-13T09:43:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : slirp4netns-1.1.8-1.module+el8.7.0+16533+44634e24. 23/31 2024-01-13T09:43:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : slirp4netns-1.2.1-1.module+el8.9.0+20326+387084d0. 24/31 2024-01-13T09:43:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : cockpit-podman-29-2.module+el8.7.0+16533+44634e24. 25/31 2024-01-13T09:43:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : crun-0.18-3.module+el8.7.0+16533+44634e24.x86_64 26/31 2024-01-13T09:43:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : skopeo-1:1.2.4-2.module+el8.7.0+16533+44634e24.x86 27/31 2024-01-13T09:43:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : toolbox-0.0.99.3-1.module+el8.7.0+16533+44634e24.x 28/31 2024-01-13T09:43:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : udica-0.2.4-1.module+el8.7.0+16533+44634e24.noarch 29/31 2024-01-13T09:43:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : buildah-1.19.9-6.module+el8.7.0+16533+44634e24.x86 30/31 2024-01-13T09:44:05.264 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 31/31 2024-01-13T09:44:05.264 INFO:teuthology.orchestra.run.smithi102.stdout:Installed products updated. 2024-01-13T09:44:05.792 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-01-13T09:44:05.792 INFO:teuthology.orchestra.run.smithi102.stdout:Downgraded: 2024-01-13T09:44:05.792 INFO:teuthology.orchestra.run.smithi102.stdout: conmon-2:2.0.26-3.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T09:44:05.792 INFO:teuthology.orchestra.run.smithi102.stdout: container-selinux-2:2.189.0-1.module+el8.7.0+16533+44634e24.noarch 2024-01-13T09:44:05.792 INFO:teuthology.orchestra.run.smithi102.stdout: containernetworking-plugins-0.9.1-1.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T09:44:05.793 INFO:teuthology.orchestra.run.smithi102.stdout: containers-common-1:1.2.4-2.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T09:44:05.793 INFO:teuthology.orchestra.run.smithi102.stdout: criu-3.15-1.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T09:44:05.793 INFO:teuthology.orchestra.run.smithi102.stdout: fuse-overlayfs-1.4.0-2.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T09:44:05.793 INFO:teuthology.orchestra.run.smithi102.stdout: libslirp-4.3.1-1.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T09:44:05.793 INFO:teuthology.orchestra.run.smithi102.stdout: podman-3.0.1-15.module+el8.7.0+18021+1e286cc8.x86_64 2024-01-13T09:44:05.793 INFO:teuthology.orchestra.run.smithi102.stdout: podman-catatonit-3.0.1-15.module+el8.7.0+18021+1e286cc8.x86_64 2024-01-13T09:44:05.793 INFO:teuthology.orchestra.run.smithi102.stdout: podman-docker-3.0.1-15.module+el8.7.0+18021+1e286cc8.noarch 2024-01-13T09:44:05.793 INFO:teuthology.orchestra.run.smithi102.stdout: runc-1.0.0-73.rc95.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T09:44:05.793 INFO:teuthology.orchestra.run.smithi102.stdout: slirp4netns-1.1.8-1.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T09:44:05.793 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-01-13T09:44:05.793 INFO:teuthology.orchestra.run.smithi102.stdout: buildah-1.19.9-6.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T09:44:05.793 INFO:teuthology.orchestra.run.smithi102.stdout: cockpit-podman-29-2.module+el8.7.0+16533+44634e24.noarch 2024-01-13T09:44:05.793 INFO:teuthology.orchestra.run.smithi102.stdout: crun-0.18-3.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T09:44:05.793 INFO:teuthology.orchestra.run.smithi102.stdout: skopeo-1:1.2.4-2.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T09:44:05.794 INFO:teuthology.orchestra.run.smithi102.stdout: toolbox-0.0.99.3-1.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T09:44:05.794 INFO:teuthology.orchestra.run.smithi102.stdout: udica-0.2.4-1.module+el8.7.0+16533+44634e24.noarch 2024-01-13T09:44:05.794 INFO:teuthology.orchestra.run.smithi102.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-13T09:44:05.794 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-01-13T09:44:05.794 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-01-13T09:44:05.967 DEBUG:teuthology.parallel:result is None 2024-01-13T09:44:10.991 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 31/31 2024-01-13T09:44:10.991 INFO:teuthology.orchestra.run.smithi084.stdout:Installed products updated. 2024-01-13T09:44:11.522 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:44:11.522 INFO:teuthology.orchestra.run.smithi084.stdout:Downgraded: 2024-01-13T09:44:11.522 INFO:teuthology.orchestra.run.smithi084.stdout: conmon-2:2.0.26-3.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T09:44:11.522 INFO:teuthology.orchestra.run.smithi084.stdout: container-selinux-2:2.189.0-1.module+el8.7.0+16533+44634e24.noarch 2024-01-13T09:44:11.522 INFO:teuthology.orchestra.run.smithi084.stdout: containernetworking-plugins-0.9.1-1.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T09:44:11.522 INFO:teuthology.orchestra.run.smithi084.stdout: containers-common-1:1.2.4-2.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T09:44:11.522 INFO:teuthology.orchestra.run.smithi084.stdout: criu-3.15-1.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T09:44:11.523 INFO:teuthology.orchestra.run.smithi084.stdout: fuse-overlayfs-1.4.0-2.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T09:44:11.523 INFO:teuthology.orchestra.run.smithi084.stdout: libslirp-4.3.1-1.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T09:44:11.523 INFO:teuthology.orchestra.run.smithi084.stdout: podman-3.0.1-15.module+el8.7.0+18021+1e286cc8.x86_64 2024-01-13T09:44:11.523 INFO:teuthology.orchestra.run.smithi084.stdout: podman-catatonit-3.0.1-15.module+el8.7.0+18021+1e286cc8.x86_64 2024-01-13T09:44:11.523 INFO:teuthology.orchestra.run.smithi084.stdout: podman-docker-3.0.1-15.module+el8.7.0+18021+1e286cc8.noarch 2024-01-13T09:44:11.523 INFO:teuthology.orchestra.run.smithi084.stdout: runc-1.0.0-73.rc95.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T09:44:11.523 INFO:teuthology.orchestra.run.smithi084.stdout: slirp4netns-1.1.8-1.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T09:44:11.523 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-01-13T09:44:11.524 INFO:teuthology.orchestra.run.smithi084.stdout: buildah-1.19.9-6.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T09:44:11.524 INFO:teuthology.orchestra.run.smithi084.stdout: cockpit-podman-29-2.module+el8.7.0+16533+44634e24.noarch 2024-01-13T09:44:11.524 INFO:teuthology.orchestra.run.smithi084.stdout: crun-0.18-3.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T09:44:11.524 INFO:teuthology.orchestra.run.smithi084.stdout: skopeo-1:1.2.4-2.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T09:44:11.524 INFO:teuthology.orchestra.run.smithi084.stdout: toolbox-0.0.99.3-1.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T09:44:11.524 INFO:teuthology.orchestra.run.smithi084.stdout: udica-0.2.4-1.module+el8.7.0+16533+44634e24.noarch 2024-01-13T09:44:11.524 INFO:teuthology.orchestra.run.smithi084.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-13T09:44:11.524 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:44:11.525 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-01-13T09:44:11.708 DEBUG:teuthology.parallel:result is None 2024-01-13T09:44:11.709 INFO:teuthology.run_tasks:Running task nvme_loop... 2024-01-13T09:44:11.719 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2024-01-13T09:44:11.719 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-01-13T09:44:11.719 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/scratch_devs of=/dev/stdout 2024-01-13T09:44:11.736 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-13T09:44:11.736 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_1 2024-01-13T09:44:11.794 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-13T09:44:11.794 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-13T09:44:11.794 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 6h/6d Inode: 171325 Links: 1 2024-01-13T09:44:11.794 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-13T09:44:11.794 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2024-01-13T09:44:11.794 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-01-13 09:43:56.094988024 +0000 2024-01-13T09:44:11.794 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-01-13 09:42:12.186017782 +0000 2024-01-13T09:44:11.794 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-01-13 09:42:12.186017782 +0000 2024-01-13T09:44:11.794 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: - 2024-01-13T09:44:11.794 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-13T09:44:11.859 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-01-13T09:44:11.859 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-01-13T09:44:11.859 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000228416 s, 2.2 MB/s 2024-01-13T09:44:11.860 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-13T09:44:11.918 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_2 2024-01-13T09:44:11.975 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-13T09:44:11.975 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-13T09:44:11.975 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 6h/6d Inode: 169206 Links: 1 2024-01-13T09:44:11.975 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-13T09:44:11.976 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2024-01-13T09:44:11.976 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-01-13 09:43:56.094988024 +0000 2024-01-13T09:44:11.976 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-01-13 09:42:12.535011642 +0000 2024-01-13T09:44:11.976 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-01-13 09:42:12.535011642 +0000 2024-01-13T09:44:11.976 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: - 2024-01-13T09:44:11.976 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-13T09:44:12.040 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-01-13T09:44:12.040 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-01-13T09:44:12.040 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000211236 s, 2.4 MB/s 2024-01-13T09:44:12.041 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-13T09:44:12.099 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_3 2024-01-13T09:44:12.156 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-13T09:44:12.156 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-13T09:44:12.156 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 6h/6d Inode: 166794 Links: 1 2024-01-13T09:44:12.156 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-13T09:44:12.156 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2024-01-13T09:44:12.156 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-01-13 09:43:56.094988024 +0000 2024-01-13T09:44:12.156 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-01-13 09:42:12.893005344 +0000 2024-01-13T09:44:12.156 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-01-13 09:42:12.893005344 +0000 2024-01-13T09:44:12.157 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: - 2024-01-13T09:44:12.157 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-13T09:44:12.221 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-01-13T09:44:12.221 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-01-13T09:44:12.221 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.00019566 s, 2.6 MB/s 2024-01-13T09:44:12.222 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-13T09:44:12.280 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_4 2024-01-13T09:44:12.337 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-13T09:44:12.337 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-13T09:44:12.337 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 6h/6d Inode: 170433 Links: 1 2024-01-13T09:44:12.337 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-13T09:44:12.337 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2024-01-13T09:44:12.337 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-01-13 09:43:56.094988024 +0000 2024-01-13T09:44:12.338 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-01-13 09:42:13.219999591 +0000 2024-01-13T09:44:12.338 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-01-13 09:42:13.219999591 +0000 2024-01-13T09:44:12.338 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: - 2024-01-13T09:44:12.338 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-13T09:44:12.402 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-01-13T09:44:12.402 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-01-13T09:44:12.402 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000213239 s, 2.4 MB/s 2024-01-13T09:44:12.403 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-13T09:44:12.461 DEBUG:teuthology.orchestra.run.smithi084:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-01-13T09:44:12.578 INFO:teuthology.orchestra.run.smithi084.stdout:loop 2024-01-13T09:44:12.580 INFO:tasks.nvme_loop:Connecting nvme_loop smithi084:/dev/vg_nvme/lv_1... 2024-01-13T09:44:12.580 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-01-13T09:44:12.615 INFO:teuthology.orchestra.run.smithi084.stdout:1 2024-01-13T09:44:12.651 INFO:teuthology.orchestra.run.smithi084.stdout:/dev/vg_nvme/lv_11 2024-01-13T09:44:12.692 INFO:tasks.nvme_loop:Connecting nvme_loop smithi084:/dev/vg_nvme/lv_2... 2024-01-13T09:44:12.692 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-01-13T09:44:12.769 INFO:teuthology.orchestra.run.smithi084.stdout:1 2024-01-13T09:44:12.804 INFO:teuthology.orchestra.run.smithi084.stdout:/dev/vg_nvme/lv_21 2024-01-13T09:44:12.843 INFO:tasks.nvme_loop:Connecting nvme_loop smithi084:/dev/vg_nvme/lv_3... 2024-01-13T09:44:12.844 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-01-13T09:44:12.917 INFO:teuthology.orchestra.run.smithi084.stdout:1 2024-01-13T09:44:12.952 INFO:teuthology.orchestra.run.smithi084.stdout:/dev/vg_nvme/lv_31 2024-01-13T09:44:12.990 INFO:tasks.nvme_loop:Connecting nvme_loop smithi084:/dev/vg_nvme/lv_4... 2024-01-13T09:44:12.991 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-01-13T09:44:13.066 INFO:teuthology.orchestra.run.smithi084.stdout:1 2024-01-13T09:44:13.101 INFO:teuthology.orchestra.run.smithi084.stdout:/dev/vg_nvme/lv_41 2024-01-13T09:44:13.139 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-01-13T09:44:13.140 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/scratch_devs of=/dev/stdout 2024-01-13T09:44:13.197 DEBUG:teuthology.orchestra.run.smithi084:> sudo nvme list 2024-01-13T09:44:13.263 INFO:teuthology.orchestra.run.smithi084.stdout:Node SN Model Namespace Usage Format FW Rev 2024-01-13T09:44:13.263 INFO:teuthology.orchestra.run.smithi084.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-01-13T09:44:13.263 INFO:teuthology.orchestra.run.smithi084.stdout:/dev/nvme0n1 PHFT620400WF400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-01-13T09:44:13.263 INFO:teuthology.orchestra.run.smithi084.stdout:/dev/nvme1n1 3cdd4f0f16ccddb21280 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2024-01-13T09:44:13.263 INFO:teuthology.orchestra.run.smithi084.stdout:/dev/nvme2n1 feb5ee0d2384fc3a127c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2024-01-13T09:44:13.263 INFO:teuthology.orchestra.run.smithi084.stdout:/dev/nvme3n1 573ce6f93c70d95e57ea Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2024-01-13T09:44:13.263 INFO:teuthology.orchestra.run.smithi084.stdout:/dev/nvme4n1 a387688e0aa28e663545 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2024-01-13T09:44:13.264 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-01-13T09:44:13.264 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-01-13T09:44:13.264 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/scratch_devs 2024-01-13T09:44:13.333 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-01-13T09:44:13.333 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/scratch_devs of=/dev/stdout 2024-01-13T09:44:13.354 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-13T09:44:13.355 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_1 2024-01-13T09:44:13.416 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-13T09:44:13.416 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-13T09:44:13.416 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 6h/6d Inode: 170115 Links: 1 2024-01-13T09:44:13.416 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-13T09:44:13.416 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-01-13T09:44:13.417 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-01-13 09:43:55.370690442 +0000 2024-01-13T09:44:13.417 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-01-13 09:42:02.482852070 +0000 2024-01-13T09:44:13.417 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-01-13 09:42:02.482852070 +0000 2024-01-13T09:44:13.417 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: - 2024-01-13T09:44:13.417 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-13T09:44:13.488 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-01-13T09:44:13.488 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-01-13T09:44:13.488 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000314727 s, 1.6 MB/s 2024-01-13T09:44:13.489 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-13T09:44:13.551 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_2 2024-01-13T09:44:13.612 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-13T09:44:13.612 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-13T09:44:13.612 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 6h/6d Inode: 168521 Links: 1 2024-01-13T09:44:13.612 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-13T09:44:13.612 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-01-13T09:44:13.613 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-01-13 09:43:55.370690442 +0000 2024-01-13T09:44:13.613 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-01-13 09:42:02.811845285 +0000 2024-01-13T09:44:13.613 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-01-13 09:42:02.811845285 +0000 2024-01-13T09:44:13.613 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: - 2024-01-13T09:44:13.613 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-13T09:44:13.683 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-01-13T09:44:13.683 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-01-13T09:44:13.683 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000284335 s, 1.8 MB/s 2024-01-13T09:44:13.684 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-13T09:44:13.746 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_3 2024-01-13T09:44:13.807 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-13T09:44:13.807 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-13T09:44:13.807 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 6h/6d Inode: 170237 Links: 1 2024-01-13T09:44:13.807 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-13T09:44:13.807 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-01-13T09:44:13.807 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-01-13 09:43:55.370690442 +0000 2024-01-13T09:44:13.807 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-01-13 09:42:03.151838272 +0000 2024-01-13T09:44:13.807 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-01-13 09:42:03.151838272 +0000 2024-01-13T09:44:13.808 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: - 2024-01-13T09:44:13.808 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-13T09:44:13.878 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-01-13T09:44:13.878 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-01-13T09:44:13.878 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000208594 s, 2.5 MB/s 2024-01-13T09:44:13.879 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-13T09:44:13.941 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_4 2024-01-13T09:44:14.002 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-13T09:44:14.002 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-13T09:44:14.002 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 6h/6d Inode: 165835 Links: 1 2024-01-13T09:44:14.002 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-13T09:44:14.003 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-01-13T09:44:14.003 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-01-13 09:43:55.370690442 +0000 2024-01-13T09:44:14.003 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-01-13 09:42:03.500831074 +0000 2024-01-13T09:44:14.003 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-01-13 09:42:03.500831074 +0000 2024-01-13T09:44:14.003 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: - 2024-01-13T09:44:14.003 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-13T09:44:14.074 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-01-13T09:44:14.074 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-01-13T09:44:14.074 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000287839 s, 1.8 MB/s 2024-01-13T09:44:14.075 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-13T09:44:14.137 DEBUG:teuthology.orchestra.run.smithi102:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-01-13T09:44:14.269 INFO:teuthology.orchestra.run.smithi102.stdout:loop 2024-01-13T09:44:14.270 INFO:tasks.nvme_loop:Connecting nvme_loop smithi102:/dev/vg_nvme/lv_1... 2024-01-13T09:44:14.271 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-01-13T09:44:14.314 INFO:teuthology.orchestra.run.smithi102.stdout:1 2024-01-13T09:44:14.355 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/vg_nvme/lv_11 2024-01-13T09:44:14.399 INFO:tasks.nvme_loop:Connecting nvme_loop smithi102:/dev/vg_nvme/lv_2... 2024-01-13T09:44:14.400 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-01-13T09:44:14.481 INFO:teuthology.orchestra.run.smithi102.stdout:1 2024-01-13T09:44:14.516 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/vg_nvme/lv_21 2024-01-13T09:44:14.556 INFO:tasks.nvme_loop:Connecting nvme_loop smithi102:/dev/vg_nvme/lv_3... 2024-01-13T09:44:14.556 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-01-13T09:44:14.635 INFO:teuthology.orchestra.run.smithi102.stdout:1 2024-01-13T09:44:14.670 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/vg_nvme/lv_31 2024-01-13T09:44:14.709 INFO:tasks.nvme_loop:Connecting nvme_loop smithi102:/dev/vg_nvme/lv_4... 2024-01-13T09:44:14.709 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-01-13T09:44:14.789 INFO:teuthology.orchestra.run.smithi102.stdout:1 2024-01-13T09:44:14.824 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/vg_nvme/lv_41 2024-01-13T09:44:14.863 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-01-13T09:44:14.863 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/scratch_devs of=/dev/stdout 2024-01-13T09:44:14.920 DEBUG:teuthology.orchestra.run.smithi102:> sudo nvme list 2024-01-13T09:44:14.985 INFO:teuthology.orchestra.run.smithi102.stdout:Node SN Model Namespace Usage Format FW Rev 2024-01-13T09:44:14.986 INFO:teuthology.orchestra.run.smithi102.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-01-13T09:44:14.986 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/nvme0n1 PHFT620400VK400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-01-13T09:44:14.986 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/nvme1n1 95a098e95c98ae499dcf Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2024-01-13T09:44:14.986 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/nvme2n1 9d81b4c10aa12b8a9868 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2024-01-13T09:44:14.986 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/nvme3n1 0ae59a8afc753da4d2ef Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2024-01-13T09:44:14.986 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/nvme4n1 556578f5419786aa9840 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2024-01-13T09:44:14.987 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-01-13T09:44:14.987 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-01-13T09:44:14.987 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/scratch_devs 2024-01-13T09:44:15.054 INFO:teuthology.run_tasks:Running task cephadm... 2024-01-13T09:44:15.123 INFO:tasks.cephadm:Config: {'roleless': True, 'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '445562ab4bc3ddfb386936119050695810860bcb'} 2024-01-13T09:44:15.123 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb 2024-01-13T09:44:15.124 INFO:tasks.cephadm:Cluster fsid is 503cf380-b1f8-11ee-95ac-87774f69a715 2024-01-13T09:44:15.124 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-01-13T09:44:15.124 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-01-13T09:44:15.124 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi084': '172.21.15.84', 'mon.smithi102': '172.21.15.102'} 2024-01-13T09:44:15.124 INFO:tasks.cephadm:Normalizing hostnames... 2024-01-13T09:44:15.124 DEBUG:teuthology.orchestra.run.smithi084:> sudo hostname $(hostname -s) 2024-01-13T09:44:15.157 DEBUG:teuthology.orchestra.run.smithi102:> sudo hostname $(hostname -s) 2024-01-13T09:44:15.184 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph-ci.git ref 445562ab4bc3ddfb386936119050695810860bcb)... 2024-01-13T09:44:15.185 DEBUG:teuthology.orchestra.run.smithi084:> curl --silent https://raw.githubusercontent.com/ceph/ceph-ci/445562ab4bc3ddfb386936119050695810860bcb/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-13T09:44:15.349 INFO:teuthology.orchestra.run.smithi084.stdout:-rw-rw-r--. 1 ubuntu ubuntu 355729 Jan 13 09:44 /home/ubuntu/cephtest/cephadm 2024-01-13T09:44:15.349 DEBUG:teuthology.orchestra.run.smithi102:> curl --silent https://raw.githubusercontent.com/ceph/ceph-ci/445562ab4bc3ddfb386936119050695810860bcb/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-13T09:44:15.487 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-rw-r--. 1 ubuntu ubuntu 355729 Jan 13 09:44 /home/ubuntu/cephtest/cephadm 2024-01-13T09:44:15.488 DEBUG:teuthology.orchestra.run.smithi084:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-13T09:44:15.509 DEBUG:teuthology.orchestra.run.smithi102:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-13T09:44:15.537 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /etc/ceph 2024-01-13T09:44:15.581 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /etc/ceph 2024-01-13T09:44:15.608 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 777 /etc/ceph 2024-01-13T09:44:15.651 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 777 /etc/ceph 2024-01-13T09:44:15.677 INFO:tasks.cephadm:Writing seed config... 2024-01-13T09:44:15.678 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-01-13T09:44:15.678 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-01-13T09:44:15.678 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-01-13T09:44:15.678 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-01-13T09:44:15.678 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-01-13T09:44:15.678 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-01-13T09:44:15.678 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-01-13T09:44:15.678 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-01-13T09:44:15.679 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-01-13T09:44:15.679 DEBUG:teuthology.orchestra.run.smithi084:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-01-13T09:44:15.710 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 503cf380-b1f8-11ee-95ac-87774f69a715 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-01-13T09:44:15.710 DEBUG:teuthology.orchestra.run.smithi084:mon.smithi084> sudo journalctl -f -n 0 -u ceph-503cf380-b1f8-11ee-95ac-87774f69a715@mon.smithi084.service 2024-01-13T09:44:15.752 INFO:tasks.cephadm:Bootstrapping... 2024-01-13T09:44:15.752 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb -v bootstrap --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.84 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-01-13T09:44:15.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:-- Logs begin at Sat 2024-01-13 09:33:34 UTC. -- 2024-01-13T09:44:16.026 INFO:teuthology.orchestra.run.smithi084.stderr:-------------------------------------------------------------------------------- 2024-01-13T09:44:16.026 INFO:teuthology.orchestra.run.smithi084.stderr:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb', '-v', 'bootstrap', '--fsid', '503cf380-b1f8-11ee-95ac-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.84', '--skip-admin-label'] 2024-01-13T09:44:16.697 INFO:teuthology.orchestra.run.smithi084.stderr:/bin/podman: 3.0.1 2024-01-13T09:44:16.711 INFO:teuthology.orchestra.run.smithi084.stderr:Verifying podman|docker is present... 2024-01-13T09:44:16.751 INFO:teuthology.orchestra.run.smithi084.stderr:/bin/podman: 3.0.1 2024-01-13T09:44:16.772 INFO:teuthology.orchestra.run.smithi084.stderr:Verifying lvm2 is present... 2024-01-13T09:44:16.773 INFO:teuthology.orchestra.run.smithi084.stderr:Verifying time synchronization is in place... 2024-01-13T09:44:16.780 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-01-13T09:44:16.785 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: inactive 2024-01-13T09:44:16.791 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: enabled 2024-01-13T09:44:16.797 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: active 2024-01-13T09:44:16.797 INFO:teuthology.orchestra.run.smithi084.stderr:Unit chronyd.service is enabled and running 2024-01-13T09:44:16.798 INFO:teuthology.orchestra.run.smithi084.stderr:Repeating the final host check... 2024-01-13T09:44:16.836 INFO:teuthology.orchestra.run.smithi084.stderr:/bin/podman: 3.0.1 2024-01-13T09:44:16.853 INFO:teuthology.orchestra.run.smithi084.stderr:podman (/bin/podman) version 3.0.1 is present 2024-01-13T09:44:16.853 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl is present 2024-01-13T09:44:16.853 INFO:teuthology.orchestra.run.smithi084.stderr:lvcreate is present 2024-01-13T09:44:16.858 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-01-13T09:44:16.864 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: inactive 2024-01-13T09:44:16.869 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: enabled 2024-01-13T09:44:16.875 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: active 2024-01-13T09:44:16.876 INFO:teuthology.orchestra.run.smithi084.stderr:Unit chronyd.service is enabled and running 2024-01-13T09:44:16.876 INFO:teuthology.orchestra.run.smithi084.stderr:Host looks OK 2024-01-13T09:44:16.876 INFO:teuthology.orchestra.run.smithi084.stderr:Cluster fsid: 503cf380-b1f8-11ee-95ac-87774f69a715 2024-01-13T09:44:16.876 INFO:teuthology.orchestra.run.smithi084.stderr:Acquiring lock 140490789429656 on /run/cephadm/503cf380-b1f8-11ee-95ac-87774f69a715.lock 2024-01-13T09:44:16.876 INFO:teuthology.orchestra.run.smithi084.stderr:Lock 140490789429656 acquired on /run/cephadm/503cf380-b1f8-11ee-95ac-87774f69a715.lock 2024-01-13T09:44:16.877 INFO:teuthology.orchestra.run.smithi084.stderr:Verifying IP 172.21.15.84 port 3300 ... 2024-01-13T09:44:16.877 INFO:teuthology.orchestra.run.smithi084.stderr:Verifying IP 172.21.15.84 port 6789 ... 2024-01-13T09:44:16.877 INFO:teuthology.orchestra.run.smithi084.stderr:Base mon IP(s) is [172.21.15.84:3300, 172.21.15.84:6789], mon addrv is [v2:172.21.15.84:3300,v1:172.21.15.84:6789] 2024-01-13T09:44:16.879 INFO:teuthology.orchestra.run.smithi084.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp metric 100 2024-01-13T09:44:16.880 INFO:teuthology.orchestra.run.smithi084.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.84 metric 100 2024-01-13T09:44:16.882 INFO:teuthology.orchestra.run.smithi084.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-01-13T09:44:16.882 INFO:teuthology.orchestra.run.smithi084.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-01-13T09:44:16.882 INFO:teuthology.orchestra.run.smithi084.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-01-13T09:44:16.885 INFO:teuthology.orchestra.run.smithi084.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-01-13T09:44:16.885 INFO:teuthology.orchestra.run.smithi084.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-01-13T09:44:16.885 INFO:teuthology.orchestra.run.smithi084.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-01-13T09:44:16.885 INFO:teuthology.orchestra.run.smithi084.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-01-13T09:44:16.885 INFO:teuthology.orchestra.run.smithi084.stderr:/sbin/ip: inet6 fe80::f084:96ca:71ac:b99/64 scope link noprefixroute 2024-01-13T09:44:16.885 INFO:teuthology.orchestra.run.smithi084.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-01-13T09:44:16.886 INFO:teuthology.orchestra.run.smithi084.stderr:Mon IP `172.21.15.84` is in CIDR network `172.21.0.0/20` 2024-01-13T09:44:16.886 INFO:teuthology.orchestra.run.smithi084.stderr:Mon IP `172.21.15.84` is in CIDR network `172.21.0.0/20` 2024-01-13T09:44:16.886 INFO:teuthology.orchestra.run.smithi084.stderr:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-01-13T09:44:16.887 INFO:teuthology.orchestra.run.smithi084.stderr:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-01-13T09:44:16.887 INFO:teuthology.orchestra.run.smithi084.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb... 2024-01-13T09:44:16.954 INFO:teuthology.orchestra.run.smithi084.stderr:/bin/podman: Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb... 2024-01-13T09:44:17.020 INFO:teuthology.orchestra.run.smithi084.stderr:/bin/podman: Getting image source signatures 2024-01-13T09:44:17.179 INFO:teuthology.orchestra.run.smithi084.stderr:/bin/podman: Copying blob sha256:9bc82cfe809169b39ceaa8362ba30ca8525e0ac28e35f12009a7a395fd81b5b0 2024-01-13T09:44:17.237 INFO:teuthology.orchestra.run.smithi084.stderr:/bin/podman: Copying blob sha256:6c507150f7e9f028ef978d633e702dab944bbda82bfbc46f6307495cf6dd9fc7 2024-01-13T09:44:32.084 INFO:teuthology.orchestra.run.smithi084.stderr:/bin/podman: Copying config sha256:163fd1944c31b631006e5d238574cf3311ed176b42b55b0b64e2387958260fc0 2024-01-13T09:44:32.134 INFO:teuthology.orchestra.run.smithi084.stderr:/bin/podman: Writing manifest to image destination 2024-01-13T09:44:32.134 INFO:teuthology.orchestra.run.smithi084.stderr:/bin/podman: Storing signatures 2024-01-13T09:44:42.064 INFO:teuthology.orchestra.run.smithi084.stderr:/bin/podman: 163fd1944c31b631006e5d238574cf3311ed176b42b55b0b64e2387958260fc0 2024-01-13T09:44:42.657 INFO:teuthology.orchestra.run.smithi084.stderr:ceph: ceph version 16.2.14-456-g445562ab (445562ab4bc3ddfb386936119050695810860bcb) pacific (stable) 2024-01-13T09:44:53.667 INFO:teuthology.orchestra.run.smithi084.stderr:Ceph version: ceph version 16.2.14-456-g445562ab (445562ab4bc3ddfb386936119050695810860bcb) pacific (stable) 2024-01-13T09:44:53.668 INFO:teuthology.orchestra.run.smithi084.stderr:Extracting ceph user uid/gid from container image... 2024-01-13T09:44:54.072 INFO:teuthology.orchestra.run.smithi084.stderr:stat: 167 167 2024-01-13T09:44:54.538 INFO:teuthology.orchestra.run.smithi084.stderr:Creating initial keys... 2024-01-13T09:44:54.963 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-authtool: AQCWW6JlYVWhOBAAamld3ibFJE+6uXURVmpZyA== 2024-01-13T09:44:55.897 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-authtool: AQCXW6JlgFI7MxAAQ+cirWzhtP5SPafbLJ9h6Q== 2024-01-13T09:44:56.737 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-authtool: AQCYW6Jlis2OKxAAaKujZKduM8lczfQfRvQScg== 2024-01-13T09:44:57.182 INFO:teuthology.orchestra.run.smithi084.stderr:Creating initial monmap... 2024-01-13T09:44:57.596 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-13T09:44:57.596 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 503cf380-b1f8-11ee-95ac-87774f69a715 2024-01-13T09:44:57.596 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-13T09:44:58.076 INFO:teuthology.orchestra.run.smithi084.stderr:monmaptool for smithi084 [v2:172.21.15.84:3300,v1:172.21.15.84:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-13T09:44:58.076 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/monmaptool: set fsid to 503cf380-b1f8-11ee-95ac-87774f69a715 2024-01-13T09:44:58.076 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-13T09:44:58.076 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-01-13T09:44:58.076 INFO:teuthology.orchestra.run.smithi084.stderr:Creating mon... 2024-01-13T09:44:58.548 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.514+0000 7f16f86f5880 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-13T09:44:58.548 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.514+0000 7f16f86f5880 1 imported monmap: 2024-01-13T09:44:58.548 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: epoch 0 2024-01-13T09:44:58.548 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: fsid 503cf380-b1f8-11ee-95ac-87774f69a715 2024-01-13T09:44:58.548 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: last_changed 2024-01-13T09:44:57.585744+0000 2024-01-13T09:44:58.548 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: created 2024-01-13T09:44:57.585744+0000 2024-01-13T09:44:58.548 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-01-13T09:44:58.548 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-01-13T09:44:58.548 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon.smithi084 2024-01-13T09:44:58.548 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-01-13T09:44:58.548 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.514+0000 7f16f86f5880 0 /usr/bin/ceph-mon: set fsid to 503cf380-b1f8-11ee-95ac-87774f69a715 2024-01-13T09:44:58.549 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: RocksDB version: 6.8.1 2024-01-13T09:44:58.549 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-01-13T09:44:58.549 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-13T09:44:58.549 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Compile date Jan 12 2024 2024-01-13T09:44:58.549 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: DB SUMMARY 2024-01-13T09:44:58.549 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-01-13T09:44:58.549 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi084/store.db dir, Total Num: 0, files: 2024-01-13T09:44:58.549 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-01-13T09:44:58.549 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi084/store.db: 2024-01-13T09:44:58.549 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-01-13T09:44:58.549 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.error_if_exists: 0 2024-01-13T09:44:58.549 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.create_if_missing: 1 2024-01-13T09:44:58.549 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.paranoid_checks: 1 2024-01-13T09:44:58.550 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.env: 0x55f882f97080 2024-01-13T09:44:58.550 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.fs: Posix File System 2024-01-13T09:44:58.550 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.info_log: 0x55f8850d7380 2024-01-13T09:44:58.550 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-13T09:44:58.550 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.statistics: (nil) 2024-01-13T09:44:58.550 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.use_fsync: 0 2024-01-13T09:44:58.550 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.max_log_file_size: 0 2024-01-13T09:44:58.550 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-13T09:44:58.550 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-13T09:44:58.550 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-13T09:44:58.550 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-13T09:44:58.550 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.allow_fallocate: 1 2024-01-13T09:44:58.550 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-13T09:44:58.550 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-13T09:44:58.550 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.use_direct_reads: 0 2024-01-13T09:44:58.550 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-13T09:44:58.551 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.create_missing_column_families: 0 2024-01-13T09:44:58.551 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.db_log_dir: 2024-01-13T09:44:58.551 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi084/store.db 2024-01-13T09:44:58.551 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-13T09:44:58.551 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.max_subcompactions: 1 2024-01-13T09:44:58.551 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.max_background_flushes: -1 2024-01-13T09:44:58.551 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-13T09:44:58.551 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-13T09:44:58.551 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-13T09:44:58.551 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-13T09:44:58.551 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-13T09:44:58.551 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.advise_random_on_open: 1 2024-01-13T09:44:58.551 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-13T09:44:58.551 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.write_buffer_manager: 0x55f8850dd980 2024-01-13T09:44:58.552 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-13T09:44:58.552 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-13T09:44:58.552 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-13T09:44:58.552 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-13T09:44:58.552 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.rate_limiter: (nil) 2024-01-13T09:44:58.552 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-13T09:44:58.552 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-13T09:44:58.552 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-13T09:44:58.552 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-13T09:44:58.552 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.unordered_write: 0 2024-01-13T09:44:58.552 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-13T09:44:58.552 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-13T09:44:58.552 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-13T09:44:58.552 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-13T09:44:58.553 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.row_cache: None 2024-01-13T09:44:58.553 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.wal_filter: None 2024-01-13T09:44:58.553 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-13T09:44:58.553 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-13T09:44:58.553 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.preserve_deletes: 0 2024-01-13T09:44:58.553 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.two_write_queues: 0 2024-01-13T09:44:58.553 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.manual_wal_flush: 0 2024-01-13T09:44:58.553 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.atomic_flush: 0 2024-01-13T09:44:58.553 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-13T09:44:58.553 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-13T09:44:58.553 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-13T09:44:58.553 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.log_readahead_size: 0 2024-01-13T09:44:58.553 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-13T09:44:58.554 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.max_background_jobs: 2 2024-01-13T09:44:58.554 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.max_background_compactions: -1 2024-01-13T09:44:58.554 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-13T09:44:58.554 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-13T09:44:58.554 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-13T09:44:58.554 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.max_total_wal_size: 0 2024-01-13T09:44:58.554 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-13T09:44:58.554 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-13T09:44:58.554 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-13T09:44:58.554 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-13T09:44:58.554 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.max_open_files: -1 2024-01-13T09:44:58.554 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.bytes_per_sync: 0 2024-01-13T09:44:58.554 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-13T09:44:58.554 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-13T09:44:58.555 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-13T09:44:58.555 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Compression algorithms supported: 2024-01-13T09:44:58.555 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-13T09:44:58.555 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: kZSTD supported: 0 2024-01-13T09:44:58.555 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: kXpressCompression supported: 0 2024-01-13T09:44:58.555 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-13T09:44:58.555 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: kLZ4Compression supported: 1 2024-01-13T09:44:58.555 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: kBZip2Compression supported: 0 2024-01-13T09:44:58.555 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: kZlibCompression supported: 1 2024-01-13T09:44:58.555 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: kSnappyCompression supported: 1 2024-01-13T09:44:58.555 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.515+0000 7f16f86f5880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-13T09:44:58.555 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-01-13T09:44:58.555 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-01-13T09:44:58.556 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi084/store.db/MANIFEST-000001 2024-01-13T09:44:58.556 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-01-13T09:44:58.556 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-13T09:44:58.556 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-01-13T09:44:58.556 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-13T09:44:58.556 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.merge_operator: 2024-01-13T09:44:58.556 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.compaction_filter: None 2024-01-13T09:44:58.556 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.compaction_filter_factory: None 2024-01-13T09:44:58.556 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-13T09:44:58.556 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-13T09:44:58.556 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f884fe7d30) 2024-01-13T09:44:58.556 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-01-13T09:44:58.556 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-13T09:44:58.556 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-13T09:44:58.557 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-01-13T09:44:58.557 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: index_type: 0 2024-01-13T09:44:58.557 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-01-13T09:44:58.557 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-01-13T09:44:58.557 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-01-13T09:44:58.557 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-01-13T09:44:58.557 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: checksum: 1 2024-01-13T09:44:58.557 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-01-13T09:44:58.557 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: block_cache: 0x55f885020d10 2024-01-13T09:44:58.557 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-01-13T09:44:58.557 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-01-13T09:44:58.557 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-01-13T09:44:58.557 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-01-13T09:44:58.557 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-01-13T09:44:58.557 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-01-13T09:44:58.557 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-01-13T09:44:58.558 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-01-13T09:44:58.558 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-01-13T09:44:58.558 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-01-13T09:44:58.558 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-01-13T09:44:58.558 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-01-13T09:44:58.558 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-01-13T09:44:58.558 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-01-13T09:44:58.558 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-01-13T09:44:58.558 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-13T09:44:58.558 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-01-13T09:44:58.558 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-01-13T09:44:58.558 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-01-13T09:44:58.558 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: format_version: 2 2024-01-13T09:44:58.558 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-01-13T09:44:58.558 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: block_align: 0 2024-01-13T09:44:58.559 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-01-13T09:44:58.559 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-13T09:44:58.559 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-13T09:44:58.559 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.compression: NoCompression 2024-01-13T09:44:58.559 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-13T09:44:58.559 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-13T09:44:58.559 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-13T09:44:58.559 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.num_levels: 7 2024-01-13T09:44:58.559 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-13T09:44:58.559 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-13T09:44:58.559 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-13T09:44:58.559 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-13T09:44:58.559 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-13T09:44:58.559 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-13T09:44:58.559 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-13T09:44:58.559 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-13T09:44:58.560 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-13T09:44:58.560 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-13T09:44:58.560 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.compression_opts.level: 32767 2024-01-13T09:44:58.560 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-13T09:44:58.560 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-13T09:44:58.560 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-13T09:44:58.560 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.compression_opts.enabled: false 2024-01-13T09:44:58.560 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-13T09:44:58.560 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-13T09:44:58.560 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-13T09:44:58.560 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-13T09:44:58.560 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-13T09:44:58.560 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-13T09:44:58.561 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-13T09:44:58.561 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-13T09:44:58.561 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-13T09:44:58.561 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-13T09:44:58.561 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-13T09:44:58.561 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-13T09:44:58.561 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-13T09:44:58.561 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-13T09:44:58.561 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-13T09:44:58.561 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-13T09:44:58.561 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-13T09:44:58.561 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.arena_block_size: 4194304 2024-01-13T09:44:58.561 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-13T09:44:58.561 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-13T09:44:58.561 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-13T09:44:58.562 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-13T09:44:58.562 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-13T09:44:58.562 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-13T09:44:58.562 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-13T09:44:58.562 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-13T09:44:58.562 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-13T09:44:58.562 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-13T09:44:58.562 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-13T09:44:58.562 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-13T09:44:58.562 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-13T09:44:58.562 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-13T09:44:58.562 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.table_properties_collectors: 2024-01-13T09:44:58.562 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.inplace_update_support: 0 2024-01-13T09:44:58.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-13T09:44:58.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-13T09:44:58.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-13T09:44:58.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-13T09:44:58.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.bloom_locality: 0 2024-01-13T09:44:58.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.max_successive_merges: 0 2024-01-13T09:44:58.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-13T09:44:58.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-13T09:44:58.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.force_consistency_checks: 0 2024-01-13T09:44:58.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-13T09:44:58.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.ttl: 2592000 2024-01-13T09:44:58.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-13T09:44:58.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi084/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-13T09:44:58.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-01-13T09:44:58.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.516+0000 7f16f86f5880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-01-13T09:44:58.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-01-13T09:44:58.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.517+0000 7f16f86f5880 4 rocksdb: DB pointer 0x55f8850eb800 2024-01-13T09:44:58.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.517+0000 7f16e156a700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-13T09:44:58.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.517+0000 7f16e156a700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-13T09:44:58.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-01-13T09:44:58.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T09:44:58.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-13T09:44:58.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-13T09:44:58.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-13T09:44:58.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-13T09:44:58.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-13T09:44:58.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-13T09:44:58.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-01-13T09:44:58.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-13T09:44:58.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T09:44:58.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T09:44:58.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-13T09:44:58.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-13T09:44:58.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-01-13T09:44:58.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-13T09:44:58.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T09:44:58.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T09:44:58.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T09:44:58.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-01-13T09:44:58.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-13T09:44:58.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-01-13T09:44:58.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-01-13T09:44:58.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-01-13T09:44:58.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T09:44:58.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T09:44:58.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-13T09:44:58.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-01-13T09:44:58.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-01-13T09:44:58.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-01-13T09:44:58.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-13T09:44:58.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T09:44:58.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T09:44:58.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-13T09:44:58.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-13T09:44:58.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-01-13T09:44:58.567 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-13T09:44:58.567 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T09:44:58.567 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T09:44:58.567 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T09:44:58.567 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-01-13T09:44:58.567 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-13T09:44:58.567 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-01-13T09:44:58.567 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-01-13T09:44:58.567 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-01-13T09:44:58.567 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T09:44:58.567 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T09:44:58.567 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-13T09:44:58.567 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-01-13T09:44:58.568 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-01-13T09:44:58.568 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-01-13T09:44:58.568 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.518+0000 7f16f86f5880 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-01-13T09:44:58.568 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.518+0000 7f16f86f5880 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-01-13T09:44:58.568 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-13T09:44:58.519+0000 7f16f86f5880 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi084 for mon.smithi084 2024-01-13T09:44:59.098 INFO:teuthology.orchestra.run.smithi084.stderr:create mon.smithi084 on 2024-01-13T09:44:59.208 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-01-13T09:44:59.304 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-503cf380-b1f8-11ee-95ac-87774f69a715.target → /etc/systemd/system/ceph-503cf380-b1f8-11ee-95ac-87774f69a715.target. 2024-01-13T09:44:59.304 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-503cf380-b1f8-11ee-95ac-87774f69a715.target → /etc/systemd/system/ceph-503cf380-b1f8-11ee-95ac-87774f69a715.target. 2024-01-13T09:44:59.506 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: Failed to reset failed state of unit ceph-503cf380-b1f8-11ee-95ac-87774f69a715@mon.smithi084.service: Unit ceph-503cf380-b1f8-11ee-95ac-87774f69a715@mon.smithi084.service not loaded. 2024-01-13T09:44:59.513 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: Created symlink /etc/systemd/system/ceph-503cf380-b1f8-11ee-95ac-87774f69a715.target.wants/ceph-503cf380-b1f8-11ee-95ac-87774f69a715@mon.smithi084.service → /etc/systemd/system/ceph-503cf380-b1f8-11ee-95ac-87774f69a715@.service. 2024-01-13T09:44:59.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:44:59 smithi084 systemd[1]: Starting Ceph mon.smithi084 for 503cf380-b1f8-11ee-95ac-87774f69a715... 2024-01-13T09:45:00.169 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: disabled 2024-01-13T09:45:00.175 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: inactive 2024-01-13T09:45:00.176 INFO:teuthology.orchestra.run.smithi084.stderr:firewalld.service is not enabled 2024-01-13T09:45:00.176 INFO:teuthology.orchestra.run.smithi084.stderr:Not possible to enable service . firewalld.service is not available 2024-01-13T09:45:00.177 INFO:teuthology.orchestra.run.smithi084.stderr:Waiting for mon to start... 2024-01-13T09:45:00.177 INFO:teuthology.orchestra.run.smithi084.stderr:Waiting for mon... 2024-01-13T09:45:00.406 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:00 smithi084 bash[32370]: bd53d600e9d206ccf966f5fcdebb6e9b147fbfa87cf96b4a28be5089bd47e12f 2024-01-13T09:45:00.407 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:00 smithi084 systemd[1]: Started Ceph mon.smithi084 for 503cf380-b1f8-11ee-95ac-87774f69a715. 2024-01-13T09:45:00.407 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:00 smithi084 conmon[32411]: cluster 2024-01-13T09:45:00.191922+0000 mon.smithi084 (mon.0) 0 : [INF] mkfs 503cf380-b1f8-11ee-95ac-87774f69a715 2024-01-13T09:45:00.407 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:00 smithi084 conmon[32411]: 2024-01-13T09:45:00.407 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:00 smithi084 conmon[32411]: cluster 2024-01-13T09:45 2024-01-13T09:45:00.407 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:00 smithi084 conmon[32411]: :00.183084+0000 mon.smithi084 (mon.0) 1 : cluster [INF] mon.smithi084 is new leader, mons smithi084 in quorum (ranks 0) 2024-01-13T09:45:00.667 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: cluster: 2024-01-13T09:45:00.667 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: id: 503cf380-b1f8-11ee-95ac-87774f69a715 2024-01-13T09:45:00.667 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-01-13T09:45:00.667 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-13T09:45:00.667 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: services: 2024-01-13T09:45:00.667 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi084 (age 0.484555s) 2024-01-13T09:45:00.667 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: mgr: no daemons active 2024-01-13T09:45:00.667 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-01-13T09:45:00.667 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-13T09:45:00.667 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: data: 2024-01-13T09:45:00.667 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-01-13T09:45:00.668 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-01-13T09:45:00.668 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-01-13T09:45:00.668 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: pgs: 2024-01-13T09:45:00.668 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-13T09:45:01.405 INFO:teuthology.orchestra.run.smithi084.stderr:mon is available 2024-01-13T09:45:01.405 INFO:teuthology.orchestra.run.smithi084.stderr:Assimilating anything we can from ceph.conf... 2024-01-13T09:45:01.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:01 smithi084 conmon[32411]: cluster 2024-01-13T09:45:00.190580+0000 mon.smithi084 (mon.0) 2 : cluster [INF] mon.smithi084 is new leader, mons smithi084 in quorum (ranks 0) 2024-01-13T09:45:01.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:01 smithi084 conmon[32411]: cluster 2024-01-13T09:45: 2024-01-13T09:45:01.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:01 smithi084 conmon[32411]: 00.190899+0000 mon.smithi084 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi084=[v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0]} removed_ranks: {} 2024-01-13T09:45:01.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:01 smithi084 conmon[32411]: cluster 2024-01-13T09:45:00.194144+0000 mon.smithi084 (mon.0) 4 : cluster [DBG] fsmap 2024-01-13T09:45:01.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:01 smithi084 conmon[32411]: cluster 2024-01-13T09:45:00.196701+0000 mon.smithi084 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-01-13T09:45:01.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:01 smithi084 conmon[32411]: cluster 2024-01-13T09:45:00.197146+0000 mon.smithi084 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-01-13T09:45:01.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:01 smithi084 conmon[32411]: audit 2024-01-13T09:45:00.675095+0000 mon.smithi084 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.84:0/2109627167' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-01-13T09:45:01.992 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-13T09:45:01.992 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: [global] 2024-01-13T09:45:01.992 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: fsid = 503cf380-b1f8-11ee-95ac-87774f69a715 2024-01-13T09:45:01.992 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.84:3300,v1:172.21.15.84:6789] 2024-01-13T09:45:01.992 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-01-13T09:45:01.992 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-01-13T09:45:01.993 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-01-13T09:45:01.993 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-01-13T09:45:01.993 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-13T09:45:01.993 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: [mgr] 2024-01-13T09:45:01.993 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-01-13T09:45:01.993 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-13T09:45:01.993 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: [osd] 2024-01-13T09:45:01.993 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-01-13T09:45:01.993 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-01-13T09:45:02.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:02 smithi084 conmon[32411]: audit 2024-01-13T09:45:01.997654+0000 mon.smithi084 (mon.0) 8 : audit [INF] from='client.? 172.21.15.84:0/1769564941' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-13T09:45:02.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:02 smithi084 conmon[32411]: audit 2024-01-13T09:45:02.000178+0000 mon.smithi084 (mon.0) 9 : audit [INF] from='client.? 172.21.15.84:0/1769564941' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-01-13T09:45:02.529 INFO:teuthology.orchestra.run.smithi084.stderr:Generating new minimal ceph.conf... 2024-01-13T09:45:03.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:03 smithi084 conmon[32411]: audit 2024-01-13T09:45:03.021067+0000 mon.smithi084 (mon.0) 2024-01-13T09:45:03.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:03 smithi084 conmon[32411]: 10 : audit [DBG] from='client.? 172.21.15.84:0/4094933498' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:45:03.538 INFO:teuthology.orchestra.run.smithi084.stderr:Restarting the monitor... 2024-01-13T09:45:03.845 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:03 smithi084 systemd[1]: Stopping Ceph mon.smithi084 for 503cf380-b1f8-11ee-95ac-87774f69a715... 2024-01-13T09:45:03.845 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:03 smithi084 bash[32916]: Error: no container with name or ID ceph-503cf380-b1f8-11ee-95ac-87774f69a715-mon.smithi084 found: no such container 2024-01-13T09:45:03.845 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:03 smithi084 conmon[32411]: debug 2024-01-13T09:45:03.853+0000 7f0ed8d48700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.smithi084 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-13T09:45:04.180 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:03 smithi084 conmon[32411]: debug 2024-01-13T09:45:03.853+0000 7f0ed8d48700 -1 mon.smithi084@0(leader) e1 *** Got Signal Terminated *** 2024-01-13T09:45:04.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:04 smithi084 bash[33031]: bd53d600e9d206ccf966f5fcdebb6e9b147fbfa87cf96b4a28be5089bd47e12f 2024-01-13T09:45:05.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:04 smithi084 systemd[1]: ceph-503cf380-b1f8-11ee-95ac-87774f69a715@mon.smithi084.service: Succeeded. 2024-01-13T09:45:05.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:04 smithi084 systemd[1]: Stopped Ceph mon.smithi084 for 503cf380-b1f8-11ee-95ac-87774f69a715. 2024-01-13T09:45:05.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:04 smithi084 systemd[1]: Starting Ceph mon.smithi084 for 503cf380-b1f8-11ee-95ac-87774f69a715... 2024-01-13T09:45:05.193 INFO:teuthology.orchestra.run.smithi084.stderr:Setting public_network to 172.21.0.0/20 in mon config section 2024-01-13T09:45:05.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.181+0000 7f0bf121b880 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-13T09:45:05.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.181+0000 7f0bf121b880 0 ceph version 16.2.14-456-g445562ab (445562ab4bc3ddfb386936119050695810860bcb) pacific (stable), process ceph-mon, pid 7 2024-01-13T09:45:05.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.181+0000 7f0bf121b880 0 pidfile_write: ignore empty --pid-file 2024-01-13T09:45:05.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.185+0000 7f0bf121b880 0 load: jerasure load: lrc load: isa 2024-01-13T09:45:05.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: RocksDB version: 6.8.1 2024-01-13T09:45:05.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: 2024-01-13T09:45:05.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-13T09:45:05.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Compile date Jan 12 2024 2024-01-13T09:45:05.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: DB SUMMARY 2024-01-13T09:45:05.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: 2024-01-13T09:45:05.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: CURRENT file: CURRENT 2024-01-13T09:45:05.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: 2024-01-13T09:45:05.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: IDENTITY file: IDENTITY 2024-01-13T09:45:05.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: 2024-01-13T09:45:05.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-01-13T09:45:05.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: 2024-01-13T09:45:05.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi084/store.db dir, Total Num: 1, files: 000004.sst 2024-01-13T09:45:05.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: 2024-01-13T09:45:05.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi084/store.db: 000006.log size: 86507 ; 2024-01-13T09:45:05.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: 2024-01-13T09:45:05.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.error_if_exists: 0 2024-01-13T09:45:05.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.create_if_missing: 0 2024-01-13T09:45:05.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.paranoid_checks: 1 2024-01-13T09:45:05.531 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.env: 0x55ada46c7080 2024-01-13T09:45:05.531 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.fs: Posix File System 2024-01-13T09:45:05.531 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.info_log: 0x55ada6aa5d20 2024-01-13T09:45:05.531 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-13T09:45:05.531 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.statistics: (nil) 2024-01-13T09:45:05.531 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.use_fsync: 0 2024-01-13T09:45:05.531 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.max_log_file_size: 0 2024-01-13T09:45:05.531 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-13T09:45:05.531 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-13T09:45:05.531 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-13T09:45:05.531 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-13T09:45:05.531 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.allow_fallocate: 1 2024-01-13T09:45:05.531 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-13T09:45:05.531 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-13T09:45:05.531 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.use_direct_reads: 0 2024-01-13T09:45:05.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-13T09:45:05.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.create_missing_column_families: 0 2024-01-13T09:45:05.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.db_log_dir: 2024-01-13T09:45:05.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi084/store.db 2024-01-13T09:45:05.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-13T09:45:05.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.max_subcompactions: 1 2024-01-13T09:45:05.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.max_background_flushes: -1 2024-01-13T09:45:05.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-13T09:45:05.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-13T09:45:05.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-13T09:45:05.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-13T09:45:05.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-13T09:45:05.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.advise_random_on_open: 1 2024-01-13T09:45:05.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-13T09:45:05.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.write_buffer_manager: 0x55ada76fc480 2024-01-13T09:45:05.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-13T09:45:05.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-13T09:45:05.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-13T09:45:05.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-13T09:45:05.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.rate_limiter: (nil) 2024-01-13T09:45:05.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-13T09:45:05.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-13T09:45:05.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-13T09:45:05.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-13T09:45:05.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.unordered_write: 0 2024-01-13T09:45:05.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-13T09:45:05.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-13T09:45:05.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-13T09:45:05.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-13T09:45:05.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.row_cache: None 2024-01-13T09:45:05.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.wal_filter: None 2024-01-13T09:45:05.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-13T09:45:05.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-13T09:45:05.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.preserve_deletes: 0 2024-01-13T09:45:05.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.two_write_queues: 0 2024-01-13T09:45:05.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.manual_wal_flush: 0 2024-01-13T09:45:05.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.atomic_flush: 0 2024-01-13T09:45:05.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-13T09:45:05.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-13T09:45:05.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-13T09:45:05.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.log_readahead_size: 0 2024-01-13T09:45:05.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-13T09:45:05.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.max_background_jobs: 2 2024-01-13T09:45:05.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.max_background_compactions: -1 2024-01-13T09:45:05.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-13T09:45:05.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-13T09:45:05.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-13T09:45:05.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.max_total_wal_size: 0 2024-01-13T09:45:05.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-13T09:45:05.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-13T09:45:05.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-13T09:45:05.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-13T09:45:05.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.max_open_files: -1 2024-01-13T09:45:05.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.bytes_per_sync: 0 2024-01-13T09:45:05.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-13T09:45:05.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-13T09:45:05.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-13T09:45:05.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Compression algorithms supported: 2024-01-13T09:45:05.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-13T09:45:05.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: kZSTD supported: 0 2024-01-13T09:45:05.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: kXpressCompression supported: 0 2024-01-13T09:45:05.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-13T09:45:05.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: kLZ4Compression supported: 1 2024-01-13T09:45:05.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: kBZip2Compression supported: 0 2024-01-13T09:45:05.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: kZlibCompression supported: 1 2024-01-13T09:45:05.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: kSnappyCompression supported: 1 2024-01-13T09:45:05.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-13T09:45:05.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi084/store.db/MANIFEST-000005 2024-01-13T09:45:05.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: 2024-01-13T09:45:05.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-13T09:45:05.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: 2024-01-13T09:45:05.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-13T09:45:05.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.merge_operator: 2024-01-13T09:45:05.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.compaction_filter: None 2024-01-13T09:45:05.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.compaction_filter_factory: None 2024-01-13T09:45:05.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-13T09:45:05.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-13T09:45:05.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ada6a4fd28) 2024-01-13T09:45:05.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: cache_index_and_filter_blocks: 1 2024-01-13T09:45:05.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-13T09:45:05.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-13T09:45:05.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: pin_top_level_index_and_filter: 1 2024-01-13T09:45:05.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: index_type: 0 2024-01-13T09:45:05.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: data_block_index_type: 0 2024-01-13T09:45:05.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: index_shortening: 1 2024-01-13T09:45:05.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: data_block_hash_table_util_ratio: 0.750000 2024-01-13T09:45:05.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: hash_index_allow_collision: 1 2024-01-13T09:45:05.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: checksum: 1 2024-01-13T09:45:05.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: no_block_cache: 0 2024-01-13T09:45:05.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: block_cache: 0x55ada6a88f10 2024-01-13T09:45:05.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: block_cache_name: BinnedLRUCache 2024-01-13T09:45:05.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: block_cache_options: 2024-01-13T09:45:05.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: capacity : 536870912 2024-01-13T09:45:05.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: num_shard_bits : 4 2024-01-13T09:45:05.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: strict_capacity_limit : 0 2024-01-13T09:45:05.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: high_pri_pool_ratio: 0.000 2024-01-13T09:45:05.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: block_cache_compressed: (nil) 2024-01-13T09:45:05.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: persistent_cache: (nil) 2024-01-13T09:45:05.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: block_size: 4096 2024-01-13T09:45:05.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: block_size_deviation: 10 2024-01-13T09:45:05.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: block_restart_interval: 16 2024-01-13T09:45:05.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: index_block_restart_interval: 1 2024-01-13T09:45:05.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: metadata_block_size: 4096 2024-01-13T09:45:05.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: partition_filters: 0 2024-01-13T09:45:05.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: use_delta_encoding: 1 2024-01-13T09:45:05.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 bash[33367]: 7e7a69d9cc893360468d0cc3b0f26dcd5023621c1a1afde5fdc2db917578ce18 2024-01-13T09:45:05.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-13T09:45:05.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: whole_key_filtering: 1 2024-01-13T09:45:05.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: verify_compression: 0 2024-01-13T09:45:05.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: read_amp_bytes_per_bit: 0 2024-01-13T09:45:05.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: format_version: 2 2024-01-13T09:45:05.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: enable_index_compression: 1 2024-01-13T09:45:05.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: block_align: 0 2024-01-13T09:45:05.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: 2024-01-13T09:45:05.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-13T09:45:05.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-13T09:45:05.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.compression: NoCompression 2024-01-13T09:45:05.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-13T09:45:05.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-13T09:45:05.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-13T09:45:05.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.num_levels: 7 2024-01-13T09:45:05.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-13T09:45:05.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-13T09:45:05.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-13T09:45:05.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-13T09:45:05.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-13T09:45:05.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-13T09:45:05.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-13T09:45:05.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-13T09:45:05.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-13T09:45:05.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-13T09:45:05.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.compression_opts.level: 32767 2024-01-13T09:45:05.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-13T09:45:05.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-13T09:45:05.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-13T09:45:05.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.compression_opts.enabled: false 2024-01-13T09:45:05.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-13T09:45:05.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-13T09:45:05.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-13T09:45:05.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-13T09:45:05.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-13T09:45:05.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-13T09:45:05.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-13T09:45:05.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-13T09:45:05.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-13T09:45:05.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-13T09:45:05.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-13T09:45:05.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-13T09:45:05.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-13T09:45:05.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-13T09:45:05.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-13T09:45:05.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-13T09:45:05.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-13T09:45:05.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.arena_block_size: 4194304 2024-01-13T09:45:05.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-13T09:45:05.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-13T09:45:05.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-13T09:45:05.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-13T09:45:05.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-13T09:45:05.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-13T09:45:05.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-13T09:45:05.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-13T09:45:05.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-13T09:45:05.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-13T09:45:05.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-13T09:45:05.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-13T09:45:05.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-13T09:45:05.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-13T09:45:05.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.table_properties_collectors: 2024-01-13T09:45:05.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.inplace_update_support: 0 2024-01-13T09:45:05.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-13T09:45:05.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-13T09:45:05.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-13T09:45:05.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-13T09:45:05.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.bloom_locality: 0 2024-01-13T09:45:05.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.max_successive_merges: 0 2024-01-13T09:45:05.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-13T09:45:05.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-13T09:45:05.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.force_consistency_checks: 0 2024-01-13T09:45:05.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-13T09:45:05.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.ttl: 2592000 2024-01-13T09:45:05.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.186+0000 7f0bf121b880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-13T09:45:05.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.187+0000 7f0bf121b880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi084/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-13T09:45:05.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: 2024-01-13T09:45:05.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.187+0000 7f0bf121b880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-01-13T09:45:05.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: 2024-01-13T09:45:05.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.187+0000 7f0bf121b880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705139105189066, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-01-13T09:45:05.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.187+0000 7f0bf121b880 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-01-13T09:45:05.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.188+0000 7f0bf121b880 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-01-13T09:45:05.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.188+0000 7f0bf121b880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705139105190245, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 83380, "table_properties": {"data_size": 81647, "index_size": 338, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 581, "raw_key_size": 9331, "raw_average_key_size": 47, "raw_value_size": 76159, "raw_average_value_size": 384, "num_data_blocks": 11, "num_entries": 198, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1705139105, "oldest_key_time": 3, "file_creation_time": 0}} 2024-01-13T09:45:05.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.188+0000 7f0bf121b880 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-01-13T09:45:05.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: 2024-01-13T09:45:05.545 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.189+0000 7f0bf121b880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705139105190792, "job": 1, "event": "recovery_finished"} 2024-01-13T09:45:05.545 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.190+0000 7f0bf121b880 4 rocksdb: DB pointer 0x55ada6b53800 2024-01-13T09:45:05.545 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.190+0000 7f0bd800f700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-13T09:45:05.545 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.190+0000 7f0bd800f700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-13T09:45:05.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: ** DB Stats ** 2024-01-13T09:45:05.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T09:45:05.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-13T09:45:05.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-13T09:45:05.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-13T09:45:05.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-13T09:45:05.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-13T09:45:05.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-13T09:45:05.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: 2024-01-13T09:45:05.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: ** Compaction Stats [default] ** 2024-01-13T09:45:05.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T09:45:05.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T09:45:05.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: L0 2/0 83.12 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 104.2 0.00 0.00 1 0.001 0 0 2024-01-13T09:45:05.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: Sum 2/0 83.12 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 104.2 0.00 0.00 1 0.001 0 0 2024-01-13T09:45:05.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 104.2 0.00 0.00 1 0.001 0 0 2024-01-13T09:45:05.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: 2024-01-13T09:45:05.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: ** Compaction Stats [default] ** 2024-01-13T09:45:05.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T09:45:05.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T09:45:05.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 104.2 0.00 0.00 1 0.001 0 0 2024-01-13T09:45:05.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T09:45:05.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-13T09:45:05.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-13T09:45:05.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: AddFile(Total Files): cumulative 0, interval 0 2024-01-13T09:45:05.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-13T09:45:05.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: AddFile(Keys): cumulative 0, interval 0 2024-01-13T09:45:05.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: Cumulative compaction: 0.00 GB write, 20.94 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T09:45:05.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: Interval compaction: 0.00 GB write, 20.94 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T09:45:05.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-13T09:45:05.548 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: 2024-01-13T09:45:05.548 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: ** File Read Latency Histogram By Level [default] ** 2024-01-13T09:45:05.548 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: 2024-01-13T09:45:05.548 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: ** Compaction Stats [default] ** 2024-01-13T09:45:05.548 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T09:45:05.548 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T09:45:05.548 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: L0 2/0 83.12 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 104.2 0.00 0.00 1 0.001 0 0 2024-01-13T09:45:05.548 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: Sum 2/0 83.12 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 104.2 0.00 0.00 1 0.001 0 0 2024-01-13T09:45:05.548 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-13T09:45:05.548 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: 2024-01-13T09:45:05.548 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: ** Compaction Stats [default] ** 2024-01-13T09:45:05.548 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T09:45:05.548 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T09:45:05.548 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 104.2 0.00 0.00 1 0.001 0 0 2024-01-13T09:45:05.548 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T09:45:05.548 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-13T09:45:05.549 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-13T09:45:05.549 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: AddFile(Total Files): cumulative 0, interval 0 2024-01-13T09:45:05.549 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-13T09:45:05.549 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: AddFile(Keys): cumulative 0, interval 0 2024-01-13T09:45:05.549 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: Cumulative compaction: 0.00 GB write, 20.79 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T09:45:05.549 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T09:45:05.549 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-13T09:45:05.549 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: 2024-01-13T09:45:05.549 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: ** File Read Latency Histogram By Level [default] ** 2024-01-13T09:45:05.549 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: 2024-01-13T09:45:05.549 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.191+0000 7f0bf121b880 0 starting mon.smithi084 rank 0 at public addrs [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] at bind addrs [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi084 fsid 503cf380-b1f8-11ee-95ac-87774f69a715 2024-01-13T09:45:05.549 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.192+0000 7f0bf121b880 1 mon.smithi084@-1(???) e1 preinit fsid 503cf380-b1f8-11ee-95ac-87774f69a715 2024-01-13T09:45:05.549 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.193+0000 7f0bf121b880 0 mon.smithi084@-1(???).mds e1 new map 2024-01-13T09:45:05.549 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.193+0000 7f0bf121b880 0 mon.smithi084@-1(???).mds e1 print_map 2024-01-13T09:45:05.550 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: e1 2024-01-13T09:45:05.550 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: enable_multiple, ever_enabled_multiple: 1,1 2024-01-13T09:45:05.550 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-13T09:45:05.550 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: legacy client fscid: -1 2024-01-13T09:45:05.550 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: 2024-01-13T09:45:05.550 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: No filesystems configured 2024-01-13T09:45:05.550 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: 2024-01-13T09:45:05.550 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.193+0000 7f0bf121b880 0 mon.smithi084@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-01-13T09:45:05.550 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.193+0000 7f0bf121b880 0 mon.smithi084@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-13T09:45:05.550 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.193+0000 7f0bf121b880 0 mon.smithi084@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-13T09:45:05.550 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.193+0000 7f0bf121b880 0 mon.smithi084@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-13T09:45:05.550 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: audit 2024-01-13T09:45:05.550 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: 2024-01-13T09: 2024-01-13T09:45:05.550 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: 45:03.021067+0000 mon.smithi084 2024-01-13T09:45:05.550 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: (mon.0) 10 : audit 2024-01-13T09:45:05.551 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: [DBG] from='client.? 172.21.15.84:0/4094933498' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:45:05.551 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: debug 2024-01-13T09:45:05.193+0000 7f0bf121b880 1 mon.smithi084@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-01-13T09:45:05.551 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 systemd[1]: Started Ceph mon.smithi084 for 503cf380-b1f8-11ee-95ac-87774f69a715. 2024-01-13T09:45:05.551 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: cluster 2024-01-13T09:45:05.211143+0000 mon.smithi084 (mon. 2024-01-13T09:45:05.551 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: 0) 1 : cluster [INF] mon.smithi084 is new leader, mons smithi084 in quorum (ranks 0) 2024-01-13T09:45:05.551 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: cluster 2024-01-13T09:45:05.211210+0000 mon.smithi084 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi084=[v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0]} removed_ranks: {} 2024-01-13T09:45:05.551 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: cluster 2024-01-13T09:45:05.211711+0000 mon.smithi084 (mon.0) 3 : cluster [DBG] fsmap 2024-01-13T09:45:05.551 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: cluster 2024-01-13T09:45:05.211743+0000 mon.smithi084 (mon. 2024-01-13T09:45:05.551 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: 0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-01-13T09:45:05.551 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: cluster 2024-01-13T09:45:05.212129+0000 2024-01-13T09:45:05.551 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:05 smithi084 conmon[33408]: mon.smithi084 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-01-13T09:45:06.304 INFO:teuthology.orchestra.run.smithi084.stderr:Wrote config to /etc/ceph/ceph.conf 2024-01-13T09:45:06.304 INFO:teuthology.orchestra.run.smithi084.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-01-13T09:45:06.304 INFO:teuthology.orchestra.run.smithi084.stderr:Creating mgr... 2024-01-13T09:45:06.305 INFO:teuthology.orchestra.run.smithi084.stderr:Verifying port 9283 ... 2024-01-13T09:45:06.435 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: Failed to reset failed state of unit ceph-503cf380-b1f8-11ee-95ac-87774f69a715@mgr.smithi084.itrhok.service: Unit ceph-503cf380-b1f8-11ee-95ac-87774f69a715@mgr.smithi084.itrhok.service not loaded. 2024-01-13T09:45:06.441 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: Created symlink /etc/systemd/system/ceph-503cf380-b1f8-11ee-95ac-87774f69a715.target.wants/ceph-503cf380-b1f8-11ee-95ac-87774f69a715@mgr.smithi084.itrhok.service → /etc/systemd/system/ceph-503cf380-b1f8-11ee-95ac-87774f69a715@.service. 2024-01-13T09:45:06.980 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:06 smithi084 conmon[33408]: audit 2024-01-13T09:45:05.693599+0000 2024-01-13T09:45:06.981 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:06 smithi084 conmon[33408]: mon.smithi084 (mon.0) 6 : audit [INF] from='client.? 172.21.15.84:0/2646003093' entity='client.admin' 2024-01-13T09:45:07.135 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: disabled 2024-01-13T09:45:07.142 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: inactive 2024-01-13T09:45:07.143 INFO:teuthology.orchestra.run.smithi084.stderr:firewalld.service is not enabled 2024-01-13T09:45:07.143 INFO:teuthology.orchestra.run.smithi084.stderr:Not possible to enable service . firewalld.service is not available 2024-01-13T09:45:07.150 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: disabled 2024-01-13T09:45:07.158 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: inactive 2024-01-13T09:45:07.159 INFO:teuthology.orchestra.run.smithi084.stderr:firewalld.service is not enabled 2024-01-13T09:45:07.159 INFO:teuthology.orchestra.run.smithi084.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-01-13T09:45:07.159 INFO:teuthology.orchestra.run.smithi084.stderr:Waiting for mgr to start... 2024-01-13T09:45:07.159 INFO:teuthology.orchestra.run.smithi084.stderr:Waiting for mgr... 2024-01-13T09:45:07.753 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-13T09:45:07.753 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: { 2024-01-13T09:45:07.753 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "fsid": "503cf380-b1f8-11ee-95ac-87774f69a715", 2024-01-13T09:45:07.753 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "health": { 2024-01-13T09:45:07.753 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-13T09:45:07.753 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "checks": {}, 2024-01-13T09:45:07.753 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "mutes": [] 2024-01-13T09:45:07.753 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-13T09:45:07.754 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-13T09:45:07.754 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "quorum": [ 2024-01-13T09:45:07.754 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 0 2024-01-13T09:45:07.754 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: ], 2024-01-13T09:45:07.754 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-13T09:45:07.754 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "smithi084" 2024-01-13T09:45:07.754 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: ], 2024-01-13T09:45:07.754 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-01-13T09:45:07.754 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "monmap": { 2024-01-13T09:45:07.754 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T09:45:07.754 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-13T09:45:07.754 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-13T09:45:07.754 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-13T09:45:07.754 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "osdmap": { 2024-01-13T09:45:07.754 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T09:45:07.755 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-13T09:45:07.755 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-13T09:45:07.755 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-13T09:45:07.758 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-13T09:45:07.758 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-13T09:45:07.758 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-13T09:45:07.758 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-13T09:45:07.759 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "pgmap": { 2024-01-13T09:45:07.759 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-13T09:45:07.759 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-13T09:45:07.759 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-13T09:45:07.759 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-13T09:45:07.759 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-13T09:45:07.759 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-13T09:45:07.759 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-13T09:45:07.759 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-13T09:45:07.759 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-13T09:45:07.759 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "fsmap": { 2024-01-13T09:45:07.759 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T09:45:07.759 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-13T09:45:07.759 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-13T09:45:07.760 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-13T09:45:07.760 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-13T09:45:07.760 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "available": false, 2024-01-13T09:45:07.760 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-13T09:45:07.760 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "modules": [ 2024-01-13T09:45:07.760 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "iostat", 2024-01-13T09:45:07.760 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "nfs", 2024-01-13T09:45:07.760 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "restful" 2024-01-13T09:45:07.760 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: ], 2024-01-13T09:45:07.760 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "services": {} 2024-01-13T09:45:07.760 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-13T09:45:07.760 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "servicemap": { 2024-01-13T09:45:07.760 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T09:45:07.760 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "modified": "2024-01-13T09:45:00.192341+0000", 2024-01-13T09:45:07.760 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "services": {} 2024-01-13T09:45:07.761 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-13T09:45:07.761 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-13T09:45:07.761 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: } 2024-01-13T09:45:08.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:07 smithi084 conmon[33408]: audit 2024-01-13T09:45:07.759959+0000 mon.smithi084 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.84:0/404645519' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-13T09:45:08.345 INFO:teuthology.orchestra.run.smithi084.stderr:mgr not available, waiting (1/15)... 2024-01-13T09:45:10.934 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-13T09:45:10.934 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: { 2024-01-13T09:45:10.934 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "fsid": "503cf380-b1f8-11ee-95ac-87774f69a715", 2024-01-13T09:45:10.934 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "health": { 2024-01-13T09:45:10.934 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-13T09:45:10.935 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "checks": {}, 2024-01-13T09:45:10.935 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "mutes": [] 2024-01-13T09:45:10.935 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-13T09:45:10.935 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-13T09:45:10.935 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "quorum": [ 2024-01-13T09:45:10.935 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 0 2024-01-13T09:45:10.935 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: ], 2024-01-13T09:45:10.935 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-13T09:45:10.935 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "smithi084" 2024-01-13T09:45:10.935 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: ], 2024-01-13T09:45:10.935 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-01-13T09:45:10.935 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "monmap": { 2024-01-13T09:45:10.935 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T09:45:10.935 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-13T09:45:10.935 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-13T09:45:10.936 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-13T09:45:10.936 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "osdmap": { 2024-01-13T09:45:10.936 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T09:45:10.936 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-13T09:45:10.936 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-13T09:45:10.936 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-13T09:45:10.936 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-13T09:45:10.936 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-13T09:45:10.936 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-13T09:45:10.936 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-13T09:45:10.936 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "pgmap": { 2024-01-13T09:45:10.936 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-13T09:45:10.936 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-13T09:45:10.936 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-13T09:45:10.936 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-13T09:45:10.937 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-13T09:45:10.937 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-13T09:45:10.937 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-13T09:45:10.937 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-13T09:45:10.937 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-13T09:45:10.937 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "fsmap": { 2024-01-13T09:45:10.937 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T09:45:10.937 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-13T09:45:10.937 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-13T09:45:10.937 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-13T09:45:10.937 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-13T09:45:10.937 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "available": false, 2024-01-13T09:45:10.937 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-13T09:45:10.937 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "modules": [ 2024-01-13T09:45:10.937 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "iostat", 2024-01-13T09:45:10.938 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "nfs", 2024-01-13T09:45:10.938 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "restful" 2024-01-13T09:45:10.938 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: ], 2024-01-13T09:45:10.938 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "services": {} 2024-01-13T09:45:10.938 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-13T09:45:10.938 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "servicemap": { 2024-01-13T09:45:10.938 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T09:45:10.938 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "modified": "2024-01-13T09:45:00.192341+0000", 2024-01-13T09:45:10.938 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "services": {} 2024-01-13T09:45:10.938 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-13T09:45:10.938 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-13T09:45:10.938 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: } 2024-01-13T09:45:11.218 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:11 smithi084 conmon[33408]: audit 2024-01-13T09:45:10.937695+0000 mon.smithi084 (mon.0) 8 : audit [DBG] 2024-01-13T09:45:11.218 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:11 smithi084 conmon[33408]: from='client.? 172.21.15.84:0/2129464591' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-13T09:45:11.561 INFO:teuthology.orchestra.run.smithi084.stderr:mgr not available, waiting (2/15)... 2024-01-13T09:45:13.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: cluster 2024-01-13T09:45:12.592284+0000 mon.smithi084 ( 2024-01-13T09:45:13.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: mon.0) 9 : cluster [INF] Activating manager daemon smithi084.itrhok 2024-01-13T09:45:13.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: cluster 2024-01-13T09:45:12.594815+0000 mon.smithi084 (mon.0) 10 : cluster [DBG] 2024-01-13T09:45:13.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: mgrmap e2: smithi084.itrhok(active, starting, since 0.00261153s) 2024-01-13T09:45:13.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: audit 2024-01-13T09:45:13.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: 2024-01-13T09:45:12. 2024-01-13T09:45:13.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: 595761+0000 mon.smithi084 (mon.0) 11 : audit [DBG] 2024-01-13T09:45:13.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: from='mgr.14100 172.21.15.84:0/1471369185' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-13T09:45:13.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: audit 2024-01-13T09:45:13.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: 2024-01-13T09:45:12.595972+0000 mon.smithi084 ( 2024-01-13T09:45:13.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.84:0/1471369185' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-13T09:45:13.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: 2024-01-13T09:45:13.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: audit 2024-01-13T09:45:12.596152+0000 mon.smithi084 (mon 2024-01-13T09:45:13.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: .0) 13 : audit [DBG] from='mgr.14100 172.21.15.84:0/1471369185' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-13T09:45:13.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: 2024-01-13T09:45:13.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: audit 2024-01-13 2024-01-13T09:45:13.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: T09:45:12.596314+0000 2024-01-13T09:45:13.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: mon.smithi084 (mon.0) 14 : audit 2024-01-13T09:45:13.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: [DBG] from='mgr.14100 172.21.15.84:0/1471369185' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mon metadata", "id": "smithi084"}]: dispatch 2024-01-13T09:45:13.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: audit 2024-01-13T09:45:13.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: 2024-01-13T09:45:12 2024-01-13T09:45:13.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: .596512+0000 mon.smithi084 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.84:0/1471369185' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mgr metadata", "who": "smithi084.itrhok", "id": "smithi084.itrhok"}]: dispatch 2024-01-13T09:45:13.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: 2024-01-13T09:45:13.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: cluster 2024-01-13T09:45:13.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: 2024-01-13T09:45:12.610133+0000 2024-01-13T09:45:13.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: mon.smithi084 (mon.0) 16 : cluster [INF] Manager daemon smithi084.itrhok is now available 2024-01-13T09:45:13.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: 2024-01-13T09:45:13.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: audit 2024 2024-01-13T09:45:13.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: -01-13T09:45:12.621363+0000 mon.smithi084 (mon. 2024-01-13T09:45:13.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: 0) 17 : audit [INF] from='mgr.14100 172.21.15.84:0/1471369185' entity='mgr.smithi084.itrhok' 2024-01-13T09:45:13.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: audit 2024-01-13T09: 2024-01-13T09:45:13.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: 45:12.625334+0000 mon.smithi084 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.84:0/1471369185' entity='mgr.smithi084.itrhok' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.itrhok/mirror_snapshot_schedule"}]: dispatch 2024-01-13T09:45:13.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: 2024-01-13T09:45:13.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: audit 2024-01 2024-01-13T09:45:13.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: -13T09:45:12.627222+0000 mon.smithi084 (mon.0) 19 2024-01-13T09:45:13.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: : audit [INF] from='mgr.14100 172.21.15.84:0/1471369185' entity='mgr.smithi084.itrhok' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.itrhok/trash_purge_schedule"}]: dispatch 2024-01-13T09:45:13.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: audit 2024-01-13T09:45:12. 2024-01-13T09:45:13.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: 630831+0000 mon.smithi084 (mon.0) 20 : audit [INF] 2024-01-13T09:45:13.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: from='mgr.14100 172.21.15.84:0/1471369185' entity='mgr.smithi084.itrhok' 2024-01-13T09:45:13.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: audit 2024 2024-01-13T09:45:13.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: -01-13T09:45:12.635815+0000 mon.smithi084 (mon.0 2024-01-13T09:45:13.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:12 smithi084 conmon[33408]: ) 21 : audit [INF] from='mgr.14100 172.21.15.84:0/1471369185' entity='mgr.smithi084.itrhok' 2024-01-13T09:45:14.225 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-13T09:45:14.226 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: { 2024-01-13T09:45:14.226 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "fsid": "503cf380-b1f8-11ee-95ac-87774f69a715", 2024-01-13T09:45:14.226 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "health": { 2024-01-13T09:45:14.226 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-13T09:45:14.226 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "checks": {}, 2024-01-13T09:45:14.226 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "mutes": [] 2024-01-13T09:45:14.226 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-13T09:45:14.226 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-13T09:45:14.226 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "quorum": [ 2024-01-13T09:45:14.226 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 0 2024-01-13T09:45:14.226 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: ], 2024-01-13T09:45:14.226 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-13T09:45:14.226 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "smithi084" 2024-01-13T09:45:14.226 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: ], 2024-01-13T09:45:14.226 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "quorum_age": 9, 2024-01-13T09:45:14.227 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "monmap": { 2024-01-13T09:45:14.227 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T09:45:14.227 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-13T09:45:14.227 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-13T09:45:14.227 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-13T09:45:14.227 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "osdmap": { 2024-01-13T09:45:14.227 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T09:45:14.227 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-13T09:45:14.227 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-13T09:45:14.227 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-13T09:45:14.231 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-13T09:45:14.231 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-13T09:45:14.231 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-13T09:45:14.231 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-13T09:45:14.231 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "pgmap": { 2024-01-13T09:45:14.231 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-13T09:45:14.231 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-13T09:45:14.231 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-13T09:45:14.231 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-13T09:45:14.231 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-13T09:45:14.231 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-13T09:45:14.231 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-13T09:45:14.231 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-13T09:45:14.231 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-13T09:45:14.232 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "fsmap": { 2024-01-13T09:45:14.232 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T09:45:14.232 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-13T09:45:14.232 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-13T09:45:14.232 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-13T09:45:14.232 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-13T09:45:14.232 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "available": true, 2024-01-13T09:45:14.232 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-13T09:45:14.232 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "modules": [ 2024-01-13T09:45:14.232 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "iostat", 2024-01-13T09:45:14.232 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "nfs", 2024-01-13T09:45:14.232 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "restful" 2024-01-13T09:45:14.232 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: ], 2024-01-13T09:45:14.232 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "services": {} 2024-01-13T09:45:14.233 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-13T09:45:14.233 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "servicemap": { 2024-01-13T09:45:14.233 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T09:45:14.233 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "modified": "2024-01-13T09:45:00.192341+0000", 2024-01-13T09:45:14.233 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "services": {} 2024-01-13T09:45:14.233 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-13T09:45:14.233 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-13T09:45:14.233 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: } 2024-01-13T09:45:14.802 INFO:teuthology.orchestra.run.smithi084.stderr:mgr is available 2024-01-13T09:45:15.008 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:14 smithi084 conmon[33408]: cluster 2024-01-13T09:45:13.598279+0000 mon.smithi084 (mon.0) 2024-01-13T09:45:15.008 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:14 smithi084 conmon[33408]: 22 : cluster [DBG] mgrmap e3: smithi084.itrhok(active, since 1.00607s) 2024-01-13T09:45:15.008 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:14 smithi084 conmon[33408]: audit 2024-01-13T09:45:14.232933+0000 mon.smithi084 (mon.0) 23 : audit [DBG] from='client.? 172.21.15.84:0/3908199621' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-13T09:45:15.430 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-13T09:45:15.430 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: [global] 2024-01-13T09:45:15.430 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: fsid = 503cf380-b1f8-11ee-95ac-87774f69a715 2024-01-13T09:45:15.430 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-01-13T09:45:15.430 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-01-13T09:45:15.430 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-01-13T09:45:15.431 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-01-13T09:45:15.431 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-13T09:45:15.431 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: [mgr] 2024-01-13T09:45:15.431 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-01-13T09:45:15.431 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-13T09:45:15.431 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: [osd] 2024-01-13T09:45:15.431 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-01-13T09:45:15.431 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-01-13T09:45:15.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:15 smithi084 conmon[33408]: cluster 2024-01-13T09:45:14.598603+0000 mon.smithi084 (mon.0) 24 2024-01-13T09:45:15.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:15 smithi084 conmon[33408]: : cluster [DBG] mgrmap e4: smithi084.itrhok(active, since 2s) 2024-01-13T09:45:15.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:15 smithi084 conmon[33408]: audit 2024-01-13T09:45:15.438134+0000 mon.smithi084 (mon.0) 25 : audit [INF] from='client.? 172.21.15.84:0/1396755085' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-13T09:45:16.041 INFO:teuthology.orchestra.run.smithi084.stderr:Enabling cephadm module... 2024-01-13T09:45:17.009 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:16 smithi084 conmon[33408]: audit 2024-01-13T09:45:16.713646+0000 mon.smithi084 (mon.0) 26 : audit [INF] from='client.? 172.21.15.84:0/2991655512' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-01-13T09:45:18.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:17 smithi084 conmon[33408]: audit 2024-01-13T09:45:16.766962 2024-01-13T09:45:18.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:17 smithi084 conmon[33408]: +0000 mon.smithi084 (mon.0) 27 : audit [INF] from='client.? 172.21.15.84:0/2991655512' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-01-13T09:45:18.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:17 smithi084 conmon[33408]: cluster 2024-01-13T09:45: 2024-01-13T09:45:18.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:17 smithi084 conmon[33408]: 16.767051+0000 mon.smithi084 (mon.0) 28 : cluster [DBG] mgrmap e5: smithi084.itrhok(active, since 4s) 2024-01-13T09:45:18.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:17 smithi084 conmon[33408]: 2024-01-13T09:45:18.054 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: { 2024-01-13T09:45:18.054 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 5, 2024-01-13T09:45:18.054 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "available": true, 2024-01-13T09:45:18.054 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "active_name": "smithi084.itrhok", 2024-01-13T09:45:18.054 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_standby": 0 2024-01-13T09:45:18.054 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: } 2024-01-13T09:45:18.592 INFO:teuthology.orchestra.run.smithi084.stderr:Waiting for the mgr to restart... 2024-01-13T09:45:18.592 INFO:teuthology.orchestra.run.smithi084.stderr:Waiting for mgr epoch 5... 2024-01-13T09:45:19.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:18 smithi084 conmon[33408]: audit 2024-01-13T09:45:18.063466+0000 mon.smithi084 (mon.0) 29 : 2024-01-13T09:45:19.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:18 smithi084 conmon[33408]: audit [DBG] from='client.? 172.21.15.84:0/2922233171' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-13T09:45:22.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:22 smithi084 conmon[33408]: cluster 2024-01-13T09:45:22.168272+0000 mon.smithi084 (mon.0) 30 : cluster [INF] Active manager daemon smithi084.itrhok restarted 2024-01-13T09:45:22.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:22 smithi084 conmon[33408]: cluster 2024-01-13T09:45:22.168487+0000 mon.smithi084 (mon.0) 31 : cluster [INF] Activating manager daemon smithi084.itrhok 2024-01-13T09:45:22.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:22 smithi084 conmon[33408]: cluster 2024-01-13T09:45:22.171197+0000 mon.smithi084 (mon.0) 32 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-01-13T09:45:22.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:22 smithi084 conmon[33408]: cluster 2024-01-13T09:45:22.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:22 smithi084 conmon[33408]: 2024-01-13T09:45:22.171297+0000 mon.smithi084 (mon.0) 33 : cluster [DBG] mgrmap e6: smithi084.itrhok(active, starting, since 0.00289121s) 2024-01-13T09:45:22.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:22 smithi084 conmon[33408]: audit 2024-01-13T09 2024-01-13T09:45:22.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:22 smithi084 conmon[33408]: :45:22.172072+0000 mon.smithi084 (mon.0) 34 : audit [DBG] from='mgr.14116 172.21.15.84:0/1153252828' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mon metadata", "id": "smithi084"}]: dispatch 2024-01-13T09:45:22.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:22 smithi084 conmon[33408]: audit 2024-01-13T09:45:22.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:22 smithi084 conmon[33408]: 2024-01-13T09:45:22.172818+0000 mon.smithi084 (mon.0) 35 : audit [DBG] from='mgr.14116 172.21.15.84:0/1153252828' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mgr metadata", "who": "smithi084.itrhok", "id": "smithi084.itrhok"}]: dispatch 2024-01-13T09:45:22.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:22 smithi084 conmon[33408]: audit 2024-01-13T09:45:22.173384+0000 mon.smithi084 (mon.0 2024-01-13T09:45:22.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:22 smithi084 conmon[33408]: ) 36 : audit [DBG] from='mgr.14116 172.21.15.84:0/1153252828' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-13T09:45:22.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:22 smithi084 conmon[33408]: audit 2024-01- 2024-01-13T09:45:22.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:22 smithi084 conmon[33408]: 13T09:45:22.173551+0000 mon.smithi084 (mon.0) 37 : audit [DBG] from='mgr.14116 172.21.15.84:0/1153252828' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-13T09:45:22.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:22 smithi084 conmon[33408]: audit 2024-01-13T09:45:22.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:22 smithi084 conmon[33408]: 2024-01-13T09:45:22.173668+0000 mon.smithi084 (mon.0) 38 : audit [DBG] from='mgr.14116 172.21.15.84:0/1153252828' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-13T09:45:22.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:22 smithi084 conmon[33408]: cluster 2024-01-13T09:45:22.186768+0000 mon.smithi084 2024-01-13T09:45:22.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:22 smithi084 conmon[33408]: (mon.0) 39 : cluster [INF] Manager daemon smithi084.itrhok is now available 2024-01-13T09:45:22.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:22 smithi084 conmon[33408]: audit 2024-01-13T09:45:22.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:22 smithi084 conmon[33408]: 2024-01-13T09:45:22.198702+0000 mon.smithi084 (mon.0) 40 : audit [INF] from='mgr.14116 172.21.15.84:0/1153252828' entity='mgr.smithi084.itrhok' 2024-01-13T09:45:23.171 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: { 2024-01-13T09:45:23.171 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-01-13T09:45:23.171 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "initialized": true 2024-01-13T09:45:23.171 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: } 2024-01-13T09:45:23.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:23 smithi084 conmon[33408]: cephadm 2024-01-13T09:45:22.196200+0000 mgr.smithi084.itrhok (mgr.14116) 1 : cephadm [INF] Found migration_current of "None". Setting to last migration. 2024-01-13T09:45:23.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:23 smithi084 conmon[33408]: audit 2024-01-13T09:45:22.283869+0000 mon.smithi084 (mon.0) 41 : audit [INF] from='mgr.14116 172.21.15.84:0/1153252828' entity='mgr.smithi084.itrhok' 2024-01-13T09:45:23.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:23 smithi084 conmon[33408]: audit 2024-01-13T09:45:22.284907+0000 mon.smithi084 (mon.0) 42 : audit [DBG] from='mgr.14116 172.21.15.84:0/1153252828' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T09:45:23.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:23 smithi084 conmon[33408]: audit 2024-01-13T09:45:22.286162+0000 mon.smithi084 (mon.0) 43 : audit [DBG] from='mgr.14116 172.21.15.84:0/1153252828' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T09:45:23.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:23 smithi084 conmon[33408]: audit 2024-01-13T09:45:22.288164+0000 mon.smithi084 (mon.0) 44 : audit [DBG] from='mgr.14116 172.21.15.84:0/1153252828' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T09:45:23.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:23 smithi084 conmon[33408]: audit 2024-01-13T09:45:22.302323+0000 mon.smithi084 (mon.0) 45 : audit [INF] from='mgr.14116 172.21.15.84:0/1153252828' entity='mgr.smithi084.itrhok' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.itrhok/mirror_snapshot_schedule"}]: dispatch 2024-01-13T09:45:23.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:23 smithi084 conmon[33408]: audit 2024-01-13T09:45:22.304153+0000 mon.smithi084 (mon.0) 46 : audit [INF] from='mgr.14116 172.21.15.84:0/1153252828' entity='mgr.smithi084.itrhok' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.itrhok/trash_purge_schedule"}]: dispatch 2024-01-13T09:45:23.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:23 smithi084 conmon[33408]: cluster 2024-01-13T09:45:23.174115+0000 mon.smithi084 (mon.0) 47 : cluster [DBG] mgrmap e7: smithi084.itrhok(active, since 1.0057s) 2024-01-13T09:45:23.717 INFO:teuthology.orchestra.run.smithi084.stderr:mgr epoch 5 is available 2024-01-13T09:45:23.717 INFO:teuthology.orchestra.run.smithi084.stderr:Setting orchestrator backend to cephadm... 2024-01-13T09:45:24.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:24 smithi084 conmon[33408]: audit 2024-01-13T09:45:23.173232+0000 mgr.smithi084.itrhok (mgr.14116) 2 : audit 2024-01-13T09:45:24.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:24 smithi084 conmon[33408]: [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-13T09:45:24.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:24 smithi084 conmon[33408]: audit 2024-01-13T09:45:23.180111+0000 mgr.smithi084.itrhok (mgr.14116) 3 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-13T09:45:25.539 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: value unchanged 2024-01-13T09:45:25.564 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:25 smithi084 conmon[33408]: cluster 2024-01-13T09:45:24.285935+0000 mon.smithi084 (mon.0 2024-01-13T09:45:25.564 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:25 smithi084 conmon[33408]: ) 48 : cluster [DBG] mgrmap e8: smithi084.itrhok(active, since 2s) 2024-01-13T09:45:25.564 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:25 smithi084 conmon[33408]: audit 2024-01-13T09:45:24.404316+0000 mgr.smithi084.itrhok (mgr.14116) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T09:45:25.564 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:25 smithi084 conmon[33408]: audit 2024-01-13T09:45:25.564 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:25 smithi084 conmon[33408]: 2024-01-13T09:45:24.408752+0000 mon.smithi084 (mon.0) 49 : audit [INF] from='mgr.14116 172.21.15.84:0/1153252828' entity='mgr.smithi084.itrhok' 2024-01-13T09:45:25.564 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:25 smithi084 conmon[33408]: audit 2024-01-13T09:45:24. 2024-01-13T09:45:25.564 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:25 smithi084 conmon[33408]: 463018+0000 mon.smithi084 (mon.0) 50 : audit [DBG] from='mgr.14116 172.21.15.84:0/1153252828' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T09:45:26.091 INFO:teuthology.orchestra.run.smithi084.stderr:Generating ssh key... 2024-01-13T09:45:26.365 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:26 smithi084 conmon[33408]: audit 2024-01-13T09:45:25.548199+0000 mgr.smithi084.itrhok (mgr.14116) 5 : audit [DBG] from='client.14128 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T09:45:27.582 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:27 smithi084 conmon[33408]: audit 2024-01-13T09:45:26. 2024-01-13T09:45:27.582 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:27 smithi084 conmon[33408]: 752213+0000 mgr.smithi084.itrhok (mgr.14116) 6 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T09:45:27.582 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:27 smithi084 conmon[33408]: cephadm 2024-01-13T09:45:26.752440+0000 mgr.smithi084.itrhok (mgr.14116) 7 : cephadm [INF] Generating ssh key... 2024-01-13T09:45:27.582 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:27 smithi084 conmon[33408]: audit 2024-01-13T09:45:26.967360+0000 mon.smithi084 (mon.0) 51 : audit [INF] from='mgr.14116 172.21.15.84:0/1153252828' entity='mgr.smithi084.itrhok' 2024-01-13T09:45:27.582 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:27 smithi084 conmon[33408]: audit 2024-01-13T09:45:26.969045+0000 mon.smithi084 (mon.0) 2024-01-13T09:45:27.582 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:27 smithi084 conmon[33408]: 52 : audit [INF] from='mgr.14116 172.21.15.84:0/1153252828' entity='mgr.smithi084.itrhok' 2024-01-13T09:45:28.184 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCe7ABt31y5j1+VIxz6Tasvlx+tpFT02H7hx0SEV7jryqe+SDXOPf4e1o/lIr5s1A7tH1IkL0hOyyhlth6COVPJTGUkQOazH2BP7dXK4aiviAd4Vzvi+4Q4U/QzENVhR9M40cHO+lVE0JtwHwcgrXFaZoxe72itN0rxVoa7HFkO553Q5zPF/hbkw+8IXarSg0kjHeTI+cF1mZCQGR0d38jtbLqEkFKDTmIJ3KtArz+8qXQepHJhe7fOu9mSpgZ3GP8YobVwXy/mMlo5yLd8Rf7mCptdKsO4R8gWCY73CorEcUlsag5mMH9f/JKkrYHGzEo9kASijl5U4CyXtnoN3r5IcN6GlDjyv7EJj3QlNt2DPmSsCtY5b1atKtQ49B89bk/efdjkMyDbD6RY0APafeSNKO74Ymo53+Fb/QWidWASiGs3xdbIKfndnjUH/H5FkuWdoTIGiLrNCkQTTXO3ZDcpdSpgezLAbHY8d2IufW0jPPridobfyUC/EpguBP08bfU= ceph-503cf380-b1f8-11ee-95ac-87774f69a715 2024-01-13T09:45:28.757 INFO:teuthology.orchestra.run.smithi084.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-01-13T09:45:28.757 INFO:teuthology.orchestra.run.smithi084.stderr:Adding key to root@localhost authorized_keys... 2024-01-13T09:45:28.758 INFO:teuthology.orchestra.run.smithi084.stderr:Adding host smithi084... 2024-01-13T09:45:29.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:28 smithi084 conmon[33408]: audit 2024-01-13T09:45:28.193728+0000 mgr.smithi084.itrhok (mgr.14116) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T09:45:29.957 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: Added host 'smithi084' with addr '172.21.15.84' 2024-01-13T09:45:30.139 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:29 smithi084 conmon[33408]: audit 2024-01-13T09:45:29.347231+0000 mgr.smithi084.itrhok (mgr.14116) 9 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi084", "addr": "172.21.15.84", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T09:45:30.606 INFO:teuthology.orchestra.run.smithi084.stderr:Deploying mon service with default placement... 2024-01-13T09:45:31.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:30 smithi084 conmon[33408]: audit 2024-01-13T09:45:29.965937+0000 mon.smithi084 (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.84:0/1153252828' entity='mgr.smithi084.itrhok' 2024-01-13T09:45:31.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:30 smithi084 conmon[33408]: cephadm 2024-01-13T09:45:29.966213+0000 mgr.smithi084.itrhok (mgr.14116) 10 : cephadm [INF] Added host smithi084 2024-01-13T09:45:31.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:30 smithi084 conmon[33408]: audit 2024-01-13T09:45:29.988834+0000 mon.smithi084 (mon.0) 54 : audit [DBG] from='mgr.14116 172.21.15.84:0/1153252828' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T09:45:31.377 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: Scheduled mon update... 2024-01-13T09:45:32.131 INFO:teuthology.orchestra.run.smithi084.stderr:Deploying mgr service with default placement... 2024-01-13T09:45:32.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:32 smithi084 conmon[33408]: audit 2024-01-13T09:45:31.382696+0000 mgr.smithi084.itrhok (mgr.14116) 11 2024-01-13T09:45:32.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:32 smithi084 conmon[33408]: : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T09:45:32.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:32 smithi084 conmon[33408]: cephadm 2024-01-13T09:45:31.383853+0000 mgr.smithi084.itrhok (mgr.14116) 12 : cephadm [INF] Saving service mon spec with placement count:5 2024-01-13T09:45:32.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:32 smithi084 conmon[33408]: audit 2024-01-13T09:45:31.386028+0000 mon.smithi084 (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.84:0/1153252828' entity='mgr.smithi084.itrhok' 2024-01-13T09:45:33.341 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-01-13T09:45:34.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:33 smithi084 conmon[33408]: audit 2024-01-13T09:45:32.908343+0000 mon.smithi084 (mon.0) 56 2024-01-13T09:45:34.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:33 smithi084 conmon[33408]: : audit [INF] from='mgr.14116 172.21.15.84:0/1153252828' entity='mgr.smithi084.itrhok' 2024-01-13T09:45:34.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:33 smithi084 conmon[33408]: audit 2024-01-13T09:45:33.179174+0000 mon.smithi084 (mon.0) 57 : audit [INF] from='mgr.14116 172.21.15.84:0/1153252828' entity='mgr.smithi084.itrhok' 2024-01-13T09:45:34.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:33 smithi084 conmon[33408]: audit 2024-01-13T09:45:33.347369+0000 mgr.smithi084.itrhok (mgr.14116) 13 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T09:45:34.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:33 smithi084 conmon[33408]: cephadm 2024-01-13T09:45:33.348188+0000 mgr.smithi084.itrhok (mgr.14116) 14 : cephadm [INF] Saving service mgr spec with placement count:2 2024-01-13T09:45:34.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:33 smithi084 conmon[33408]: audit 2024-01-13T09:45:33.350370+0000 mon.smithi084 (mon.0) 58 : audit [INF] from='mgr.14116 172.21.15.84:0/1153252828' entity='mgr.smithi084.itrhok' 2024-01-13T09:45:34.047 INFO:teuthology.orchestra.run.smithi084.stderr:Deploying crash service with default placement... 2024-01-13T09:45:34.691 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: Scheduled crash update... 2024-01-13T09:45:35.296 INFO:teuthology.orchestra.run.smithi084.stderr:Deploying prometheus service with default placement... 2024-01-13T09:45:35.897 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:35 smithi084 conmon[33408]: audit 2024-01-13T09:45:34.697276+0000 mgr.smithi084.itrhok (mgr.14116) 15 : audit [DBG] from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T09:45:35.897 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:35 smithi084 conmon[33408]: cephadm 2024-01-13T09:45:34.698214+0000 mgr.smithi084.itrhok (mgr.14116) 16 : cephadm [INF] Saving service crash spec with placement * 2024-01-13T09:45:35.897 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:35 smithi084 conmon[33408]: audit 2024-01-13T09:45:34.700311+0000 mon.smithi084 (mon.0) 59 : audit [INF] from='mgr.14116 172.21.15.84:0/1153252828' entity='mgr.smithi084.itrhok' 2024-01-13T09:45:35.897 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:35 smithi084 conmon[33408]: audit 2024-01-13T09:45:34.794452+0000 mon.smithi084 (mon.0) 60 : audit [INF] from='mgr.14116 172.21.15.84:0/1153252828' entity='mgr.smithi084.itrhok' 2024-01-13T09:45:36.018 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-01-13T09:45:37.275 INFO:teuthology.orchestra.run.smithi084.stderr:Deploying grafana service with default placement... 2024-01-13T09:45:37.275 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:37 smithi084 conmon[33408]: audit 2024-01-13T09:45:36.025173+0000 mgr.smithi084.itrhok (mgr.14116) 17 : audit 2024-01-13T09:45:37.275 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:37 smithi084 conmon[33408]: [DBG] from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T09:45:37.275 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:37 smithi084 conmon[33408]: cephadm 2024-01-13T09:45:36.026046+0000 mgr.smithi084.itrhok (mgr.14116) 18 : cephadm [INF] Saving service prometheus spec with placement count:1 2024-01-13T09:45:37.275 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:37 smithi084 conmon[33408]: audit 2024-01-13T09:45:36.028146+0000 mon.smithi084 (mon.0) 61 : audit [INF] from='mgr.14116 172.21.15.84:0/1153252828' entity='mgr.smithi084.itrhok' 2024-01-13T09:45:37.973 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-01-13T09:45:38.521 INFO:teuthology.orchestra.run.smithi084.stderr:Deploying node-exporter service with default placement... 2024-01-13T09:45:39.122 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-01-13T09:45:39.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:38 smithi084 conmon[33408]: audit 2024-01-13T09:45:37.979806+0000 mgr.smithi084.itrhok (mgr.14116) 19 : audit [DBG] from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T09:45:39.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:38 smithi084 conmon[33408]: cephadm 2024-01-13T09:45:37.980638+0000 mgr.smithi084.itrhok (mgr.14116) 20 : cephadm [INF] Saving service grafana spec with placement count:1 2024-01-13T09:45:39.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:38 smithi084 conmon[33408]: audit 2024-01-13T09:45:37.982740+0000 mon.smithi084 (mon.0) 62 : audit [INF] from='mgr.14116 172.21.15.84:0/1153252828' entity='mgr.smithi084.itrhok' 2024-01-13T09:45:39.696 INFO:teuthology.orchestra.run.smithi084.stderr:Deploying alertmanager service with default placement... 2024-01-13T09:45:40.340 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-01-13T09:45:40.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:40 smithi084 conmon[33408]: audit 2024-01-13T09:45:39.128203+0000 mgr.smithi084.itrhok (mgr.14116) 21 : audit [DBG] from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T09:45:40.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:40 smithi084 conmon[33408]: cephadm 2024-01-13T09:45:39.128996+0000 mgr.smithi084.itrhok (mgr.14116) 22 : cephadm [INF] Saving service node-exporter spec with placement * 2024-01-13T09:45:40.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:40 smithi084 conmon[33408]: audit 2024 2024-01-13T09:45:40.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:40 smithi084 conmon[33408]: -01-13T09:45:39.131871+0000 mon.smithi084 (mon 2024-01-13T09:45:40.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:40 smithi084 conmon[33408]: .0) 63 : audit [INF] from='mgr.14116 172.21.15.84:0/1153252828' entity='mgr.smithi084.itrhok' 2024-01-13T09:45:41.720 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:41 smithi084 conmon[33408]: audit 2024-01-13T09:45:40.347235+0000 mgr.smithi084.itrhok (mgr.14116) 23 : audit [DBG] 2024-01-13T09:45:41.720 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:41 smithi084 conmon[33408]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T09:45:41.720 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:41 smithi084 conmon[33408]: cephadm 2024-01-13T09:45:40.348063+0000 mgr.smithi084.itrhok (mgr.14116) 24 : cephadm [INF] Saving service alertmanager spec with placement count:1 2024-01-13T09:45:41.720 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:41 smithi084 conmon[33408]: audit 2024-01-13T09:45:40.350271+0000 mon.smithi084 (mon.0) 64 : audit [INF] from='mgr.14116 172.21.15.84:0/1153252828' entity='mgr.smithi084.itrhok' 2024-01-13T09:45:42.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:42 smithi084 conmon[33408]: audit 2024-01-13T09:45:41.503292+0000 mon.smithi084 (mon.0) 65 : audit [INF] from='client.? 172.21.15.84:0/103672304' entity='client.admin' 2024-01-13T09:45:43.262 INFO:teuthology.orchestra.run.smithi084.stderr:Enabling the dashboard module... 2024-01-13T09:45:44.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:43 smithi084 conmon[33408]: cluster 2024-01-13T09:45:42.174682+0000 mgr.smithi084.itrhok (mgr.14116) 25 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:45:44.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:43 smithi084 conmon[33408]: audit 2024-01-13T09:45:42.727838+0000 mon.smithi084 (mon.0) 66 : audit [INF] from='client.? 172.21.15.84:0/3575966603' entity='client.admin' 2024-01-13T09:45:44.977 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:44 smithi084 conmon[33408]: audit 2024-01-13T09:45:43.916107+0000 mon.smithi084 (mon.0) 67 : audit [INF] from='client.? 172.21.15.84:0/2409185842' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-01-13T09:45:46.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:45 smithi084 conmon[33408]: audit 2024-01-13T09:45:44.733029+0000 mon.smithi084 (mon 2024-01-13T09:45:46.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:45 smithi084 conmon[33408]: .0) 68 : audit [INF] from='client.? 172.21.15.84:0/2409185842' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-01-13T09:45:46.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:45 smithi084 conmon[33408]: cluster 2024-01-13T09:45:44.733117+0000 mon.smithi084 (mon.0) 69 : cluster [DBG] mgrmap e9: smithi084.itrhok(active, since 22s) 2024-01-13T09:45:46.119 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: { 2024-01-13T09:45:46.119 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 9, 2024-01-13T09:45:46.120 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "available": true, 2024-01-13T09:45:46.120 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "active_name": "smithi084.itrhok", 2024-01-13T09:45:46.120 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_standby": 0 2024-01-13T09:45:46.120 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: } 2024-01-13T09:45:46.745 INFO:teuthology.orchestra.run.smithi084.stderr:Waiting for the mgr to restart... 2024-01-13T09:45:46.746 INFO:teuthology.orchestra.run.smithi084.stderr:Waiting for mgr epoch 9... 2024-01-13T09:45:46.980 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:46 smithi084 conmon[33408]: audit 2024-01-13T09:45:46.126137+0000 mon.smithi084 (mon.0 2024-01-13T09:45:46.980 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:46 smithi084 conmon[33408]: ) 70 : audit [DBG] from='client.? 172.21.15.84:0/3665456372' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-13T09:45:50.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:50 smithi084 conmon[33408]: cluster 2024-01-13T09:45:50.189371+0000 mon.smithi084 (mon.0 2024-01-13T09:45:50.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:50 smithi084 conmon[33408]: ) 71 : cluster [INF] Active manager daemon smithi084.itrhok restarted 2024-01-13T09:45:50.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:50 smithi084 conmon[33408]: cluster 2024-01-13T09:45:50.189589+0000 mon.smithi084 (mon.0) 72 : cluster [INF] Activating manager daemon smithi084.itrhok 2024-01-13T09:45:50.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:50 smithi084 conmon[33408]: cluster 2024-01-13T09:45 2024-01-13T09:45:50.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:50 smithi084 conmon[33408]: :50.192296+0000 mon.smithi084 (mon.0) 73 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-01-13T09:45:50.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:50 smithi084 conmon[33408]: cluster 2024-01-13T09:45:50. 2024-01-13T09:45:50.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:50 smithi084 conmon[33408]: 192404+0000 mon.smithi084 (mon.0) 74 : cluster [DBG] mgrmap e10: smithi084.itrhok(active, starting, since 0.00289941s) 2024-01-13T09:45:50.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:50 smithi084 conmon[33408]: audit 2024-01-13T09:45:50.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:50 smithi084 conmon[33408]: 2024-01-13T09:45:50.193083+0000 mon.smithi084 (mon.0) 75 : audit [DBG] from='mgr.14156 172.21.15.84:0/355417686' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mon metadata", "id": "smithi084"}]: dispatch 2024-01-13T09:45:50.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:50 smithi084 conmon[33408]: audit 2024-01-13T09:45:50. 2024-01-13T09:45:50.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:50 smithi084 conmon[33408]: 193712+0000 mon.smithi084 (mon.0) 76 : audit [DBG] from='mgr.14156 172.21.15.84:0/355417686' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mgr metadata", "who": "smithi084.itrhok", "id": "smithi084.itrhok"}]: dispatch 2024-01-13T09:45:50.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:50 smithi084 conmon[33408]: audit 2024-01-13T09:45:50.194546+0000 mon.smithi084 (mon.0) 2024-01-13T09:45:50.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:50 smithi084 conmon[33408]: 77 : audit [DBG] from='mgr.14156 172.21.15.84:0/355417686' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-13T09:45:50.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:50 smithi084 conmon[33408]: audit 2024-01-13T09:45:50.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:50 smithi084 conmon[33408]: 2024-01-13T09:45:50.194699+0000 mon.smithi084 (mon.0) 78 : audit [DBG] from='mgr.14156 172.21.15.84:0/355417686' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-13T09:45:50.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:50 smithi084 conmon[33408]: audit 2024-01-13T09:45:50. 2024-01-13T09:45:50.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:50 smithi084 conmon[33408]: 194847+0000 mon.smithi084 (mon.0) 79 : audit [DBG] from='mgr.14156 172.21.15.84:0/355417686' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-13T09:45:50.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:50 smithi084 conmon[33408]: cluster 2024-01-13T09:45:50.208876+0000 mon.smithi084 2024-01-13T09:45:50.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:50 smithi084 conmon[33408]: (mon.0) 80 : cluster [INF] Manager daemon smithi084.itrhok is now available 2024-01-13T09:45:51.205 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: { 2024-01-13T09:45:51.205 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-01-13T09:45:51.205 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "initialized": true 2024-01-13T09:45:51.205 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: } 2024-01-13T09:45:51.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:51 smithi084 conmon[33408]: audit 2024-01-13T09:45:50.304728+0000 mon.smithi084 (mon.0) 81 : audit [DBG] from='mgr.14156 172.21.15.84:0/355417686' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T09:45:51.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:51 smithi084 conmon[33408]: audit 2024-01-13T09:45:50.306115+0000 mon.smithi084 (mon.0) 82 : audit [DBG] from='mgr.14156 172.21.15.84:0/355417686' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T09:45:51.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:51 smithi084 conmon[33408]: audit 2024-01-13T09:45:50.330739+0000 mon.smithi084 (mon.0) 83 : audit [INF] from='mgr.14156 172.21.15.84:0/355417686' entity='mgr.smithi084.itrhok' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.itrhok/mirror_snapshot_schedule"}]: dispatch 2024-01-13T09:45:51.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:51 smithi084 conmon[33408]: audit 2024-01-13T09:45:50.333601+0000 mon.smithi084 (mon.0) 84 : audit [INF] from='mgr.14156 172.21.15.84:0/355417686' entity='mgr.smithi084.itrhok' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.itrhok/trash_purge_schedule"}]: dispatch 2024-01-13T09:45:51.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:51 smithi084 conmon[33408]: cluster 2024-01-13T09:45:51.209159+0000 mon.smithi084 (mon.0) 85 : cluster [DBG] mgrmap e11: smithi084.itrhok(active, since 1.01965s) 2024-01-13T09:45:51.885 INFO:teuthology.orchestra.run.smithi084.stderr:mgr epoch 9 is available 2024-01-13T09:45:51.885 INFO:teuthology.orchestra.run.smithi084.stderr:Generating a dashboard self-signed certificate... 2024-01-13T09:45:52.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:52 smithi084 conmon[33408]: audit 2024-01-13T09:45:51.208569+0000 mgr.smithi084.itrhok (mgr.14156) 1 : audit [DBG] from='client.14160 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-13T09:45:52.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:52 smithi084 conmon[33408]: audit 2024-01-13T09 2024-01-13T09:45:52.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:52 smithi084 conmon[33408]: :45:51.214644+0000 mgr.smithi084.itrhok (mgr.14156) 2 : audit [DBG] from='client.14160 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-13T09:45:52.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:52 smithi084 conmon[33408]: audit 2024-01-13T09:45:51.841986+0000 mon.smithi084 (mon.0) 86 : audit [INF] from='mgr.14156 172.21.15.84:0/355417686' entity='mgr.smithi084.itrhok' 2024-01-13T09:45:52.622 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: Self-signed certificate created 2024-01-13T09:45:53.530 INFO:teuthology.orchestra.run.smithi084.stderr:Creating initial admin user... 2024-01-13T09:45:53.916 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:53 smithi084 conmon[33408]: audit 2024-01-13T09:45:52.591598+0000 mgr.smithi084.itrhok (mgr.14156) 3 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T09:45:53.916 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:53 smithi084 conmon[33408]: audit 2024-01-13T09:45:52.630181+0000 mon.smithi084 (mon.0) 87 : audit [INF] from='mgr.14156 172.21.15.84:0/355417686' entity='mgr.smithi084.itrhok' 2024-01-13T09:45:53.916 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:53 smithi084 conmon[33408]: audit 2024-01-13T09:45:52.632075+0000 mon.smithi084 (mon.0) 88 : audit [INF] from='mgr.14156 172.21.15.84:0/355417686' entity='mgr.smithi084.itrhok' 2024-01-13T09:45:53.917 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:53 smithi084 conmon[33408]: cluster 2024-01-13T09:45:52.843035+0000 mon.smithi084 (mon.0) 89 : cluster [DBG] mgrmap e12: smithi084.itrhok(active, since 2s) 2024-01-13T09:45:54.538 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$3..2ywTfO.sTvGezHURzg.0Y3lXZC6JD34CQNXNRHK0OAsALUvsq.", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1705139154, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-01-13T09:45:55.151 INFO:teuthology.orchestra.run.smithi084.stderr:Fetching dashboard port number... 2024-01-13T09:45:55.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:55 smithi084 conmon[33408]: audit 2024-01-13T09:45:54.331516+0000 mgr.smithi084.itrhok (mgr.14156 2024-01-13T09:45:55.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:55 smithi084 conmon[33408]: ) 4 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T09:45:55.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:55 smithi084 conmon[33408]: audit 2024-01-13T09:45:54.547888+0000 mon.smithi084 (mon.0) 90 : audit [INF] from='mgr.14156 172.21.15.84:0/355417686' entity='mgr.smithi084.itrhok' 2024-01-13T09:45:55.831 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 8443 2024-01-13T09:45:56.535 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: disabled 2024-01-13T09:45:56.542 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: inactive 2024-01-13T09:45:56.543 INFO:teuthology.orchestra.run.smithi084.stderr:firewalld.service is not enabled 2024-01-13T09:45:56.543 INFO:teuthology.orchestra.run.smithi084.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-01-13T09:45:56.545 INFO:teuthology.orchestra.run.smithi084.stderr:Ceph Dashboard is now available at: 2024-01-13T09:45:56.545 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-01-13T09:45:56.545 INFO:teuthology.orchestra.run.smithi084.stderr: URL: https://smithi084:8443/ 2024-01-13T09:45:56.546 INFO:teuthology.orchestra.run.smithi084.stderr: User: admin 2024-01-13T09:45:56.546 INFO:teuthology.orchestra.run.smithi084.stderr: Password: rp3ewqb7zp 2024-01-13T09:45:56.546 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-01-13T09:45:56.546 INFO:teuthology.orchestra.run.smithi084.stderr:Enabling autotune for osd_memory_target 2024-01-13T09:45:56.800 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:56 smithi084 conmon[33408]: audit 2024-01-13T09:45:55.841296+0000 mon.smithi084 (mon.0) 91 : audit [DBG] from='client.? 172.21.15.84:0/4056261339' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-01-13T09:45:57.915 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:57 smithi084 conmon[33408]: cluster 2024-01-13T09:45:56.552646+0000 mon.smithi084 (mon.0) 92 : cluster 2024-01-13T09:45:57.915 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:57 smithi084 conmon[33408]: [DBG] mgrmap e13: smithi084.itrhok(active, since 6s) 2024-01-13T09:45:58.423 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-01-13T09:45:58.996 INFO:teuthology.orchestra.run.smithi084.stderr:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-01-13T09:45:58.997 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-01-13T09:45:58.997 INFO:teuthology.orchestra.run.smithi084.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-01-13T09:45:58.997 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-01-13T09:45:58.997 INFO:teuthology.orchestra.run.smithi084.stderr:Or, if you are only running a single cluster on this host: 2024-01-13T09:45:58.997 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-01-13T09:45:58.997 INFO:teuthology.orchestra.run.smithi084.stderr: sudo /home/ubuntu/cephtest/cephadm shell 2024-01-13T09:45:58.997 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-01-13T09:45:58.997 INFO:teuthology.orchestra.run.smithi084.stderr:Please consider enabling telemetry to help improve Ceph: 2024-01-13T09:45:58.997 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-01-13T09:45:58.997 INFO:teuthology.orchestra.run.smithi084.stderr: ceph telemetry on 2024-01-13T09:45:58.997 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-01-13T09:45:58.997 INFO:teuthology.orchestra.run.smithi084.stderr:For more information see: 2024-01-13T09:45:58.997 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-01-13T09:45:58.998 INFO:teuthology.orchestra.run.smithi084.stderr: https://docs.ceph.com/en/pacific/mgr/telemetry/ 2024-01-13T09:45:58.998 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-01-13T09:45:58.998 INFO:teuthology.orchestra.run.smithi084.stderr:Bootstrap complete. 2024-01-13T09:45:59.055 INFO:tasks.cephadm:Fetching config... 2024-01-13T09:45:59.056 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-01-13T09:45:59.056 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-01-13T09:45:59.074 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-01-13T09:45:59.074 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-01-13T09:45:59.074 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-01-13T09:45:59.131 INFO:tasks.cephadm:Fetching mon keyring... 2024-01-13T09:45:59.131 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-01-13T09:45:59.131 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi084/keyring of=/dev/stdout 2024-01-13T09:45:59.198 INFO:tasks.cephadm:Fetching pub ssh key... 2024-01-13T09:45:59.198 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-01-13T09:45:59.198 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-01-13T09:45:59.255 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-01-13T09:45:59.255 DEBUG:teuthology.orchestra.run.smithi084:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCe7ABt31y5j1+VIxz6Tasvlx+tpFT02H7hx0SEV7jryqe+SDXOPf4e1o/lIr5s1A7tH1IkL0hOyyhlth6COVPJTGUkQOazH2BP7dXK4aiviAd4Vzvi+4Q4U/QzENVhR9M40cHO+lVE0JtwHwcgrXFaZoxe72itN0rxVoa7HFkO553Q5zPF/hbkw+8IXarSg0kjHeTI+cF1mZCQGR0d38jtbLqEkFKDTmIJ3KtArz+8qXQepHJhe7fOu9mSpgZ3GP8YobVwXy/mMlo5yLd8Rf7mCptdKsO4R8gWCY73CorEcUlsag5mMH9f/JKkrYHGzEo9kASijl5U4CyXtnoN3r5IcN6GlDjyv7EJj3QlNt2DPmSsCtY5b1atKtQ49B89bk/efdjkMyDbD6RY0APafeSNKO74Ymo53+Fb/QWidWASiGs3xdbIKfndnjUH/H5FkuWdoTIGiLrNCkQTTXO3ZDcpdSpgezLAbHY8d2IufW0jPPridobfyUC/EpguBP08bfU= ceph-503cf380-b1f8-11ee-95ac-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-13T09:45:59.333 INFO:teuthology.orchestra.run.smithi084.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCe7ABt31y5j1+VIxz6Tasvlx+tpFT02H7hx0SEV7jryqe+SDXOPf4e1o/lIr5s1A7tH1IkL0hOyyhlth6COVPJTGUkQOazH2BP7dXK4aiviAd4Vzvi+4Q4U/QzENVhR9M40cHO+lVE0JtwHwcgrXFaZoxe72itN0rxVoa7HFkO553Q5zPF/hbkw+8IXarSg0kjHeTI+cF1mZCQGR0d38jtbLqEkFKDTmIJ3KtArz+8qXQepHJhe7fOu9mSpgZ3GP8YobVwXy/mMlo5yLd8Rf7mCptdKsO4R8gWCY73CorEcUlsag5mMH9f/JKkrYHGzEo9kASijl5U4CyXtnoN3r5IcN6GlDjyv7EJj3QlNt2DPmSsCtY5b1atKtQ49B89bk/efdjkMyDbD6RY0APafeSNKO74Ymo53+Fb/QWidWASiGs3xdbIKfndnjUH/H5FkuWdoTIGiLrNCkQTTXO3ZDcpdSpgezLAbHY8d2IufW0jPPridobfyUC/EpguBP08bfU= ceph-503cf380-b1f8-11ee-95ac-87774f69a715 2024-01-13T09:45:59.344 DEBUG:teuthology.orchestra.run.smithi102:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCe7ABt31y5j1+VIxz6Tasvlx+tpFT02H7hx0SEV7jryqe+SDXOPf4e1o/lIr5s1A7tH1IkL0hOyyhlth6COVPJTGUkQOazH2BP7dXK4aiviAd4Vzvi+4Q4U/QzENVhR9M40cHO+lVE0JtwHwcgrXFaZoxe72itN0rxVoa7HFkO553Q5zPF/hbkw+8IXarSg0kjHeTI+cF1mZCQGR0d38jtbLqEkFKDTmIJ3KtArz+8qXQepHJhe7fOu9mSpgZ3GP8YobVwXy/mMlo5yLd8Rf7mCptdKsO4R8gWCY73CorEcUlsag5mMH9f/JKkrYHGzEo9kASijl5U4CyXtnoN3r5IcN6GlDjyv7EJj3QlNt2DPmSsCtY5b1atKtQ49B89bk/efdjkMyDbD6RY0APafeSNKO74Ymo53+Fb/QWidWASiGs3xdbIKfndnjUH/H5FkuWdoTIGiLrNCkQTTXO3ZDcpdSpgezLAbHY8d2IufW0jPPridobfyUC/EpguBP08bfU= ceph-503cf380-b1f8-11ee-95ac-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-13T09:45:59.401 INFO:teuthology.orchestra.run.smithi102.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCe7ABt31y5j1+VIxz6Tasvlx+tpFT02H7hx0SEV7jryqe+SDXOPf4e1o/lIr5s1A7tH1IkL0hOyyhlth6COVPJTGUkQOazH2BP7dXK4aiviAd4Vzvi+4Q4U/QzENVhR9M40cHO+lVE0JtwHwcgrXFaZoxe72itN0rxVoa7HFkO553Q5zPF/hbkw+8IXarSg0kjHeTI+cF1mZCQGR0d38jtbLqEkFKDTmIJ3KtArz+8qXQepHJhe7fOu9mSpgZ3GP8YobVwXy/mMlo5yLd8Rf7mCptdKsO4R8gWCY73CorEcUlsag5mMH9f/JKkrYHGzEo9kASijl5U4CyXtnoN3r5IcN6GlDjyv7EJj3QlNt2DPmSsCtY5b1atKtQ49B89bk/efdjkMyDbD6RY0APafeSNKO74Ymo53+Fb/QWidWASiGs3xdbIKfndnjUH/H5FkuWdoTIGiLrNCkQTTXO3ZDcpdSpgezLAbHY8d2IufW0jPPridobfyUC/EpguBP08bfU= ceph-503cf380-b1f8-11ee-95ac-87774f69a715 2024-01-13T09:45:59.414 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-01-13T09:45:59.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:45:59 smithi084 conmon[33408]: audit 2024-01-13T09:45:58.430725+0000 mon.smithi084 (mon.0) 93 : audit [INF] from='client.? 172.21.15.84:0/214000375' entity='client.admin' 2024-01-13T09:46:02.464 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-01-13T09:46:02.464 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-01-13T09:46:03.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:02 smithi084 conmon[33408]: audit 2024-01-13T09:46:01.824472+0000 mon.smithi084 (mon.0) 94 : audit [INF] from='client.? 172.21.15.84:0/3219080240' entity='client.admin' 2024-01-13T09:46:04.981 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:04 smithi084 conmon[33408]: audit 2024-01-13T09:46:03.640509+0000 mon.smithi084 (mon.0) 95 : audit [INF] from='mgr.14156 172.21.15.84:0/355417686' entity='mgr.smithi084.itrhok' 2024-01-13T09:46:04.981 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:04 smithi084 conmon[33408]: audit 2024-01-13T09:46:04.091972+0000 mon.smithi084 (mon.0) 96 : audit [INF] from='mgr.14156 172.21.15.84:0/355417686' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config rm", "who": "osd/host:smithi084", "name": "osd_memory_target"}]: dispatch 2024-01-13T09:46:04.981 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:04 smithi084 conmon[33408]: audit 2024-01-13T09:46:04.097658+0000 mon.smithi084 (mon.0) 97 : audit [INF] from='mgr.14156 172.21.15.84:0/355417686' entity='mgr.smithi084.itrhok' 2024-01-13T09:46:05.431 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi102 2024-01-13T09:46:05.431 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-01-13T09:46:05.431 DEBUG:teuthology.orchestra.run.smithi102:> dd of=/etc/ceph/ceph.conf 2024-01-13T09:46:05.452 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-01-13T09:46:05.452 DEBUG:teuthology.orchestra.run.smithi102:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-13T09:46:05.512 INFO:tasks.cephadm:Adding host smithi102 to orchestrator... 2024-01-13T09:46:05.512 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph orch host add smithi102 2024-01-13T09:46:06.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:05 smithi084 conmon[33408]: cephadm 2024-01-13T09:46:04.105825+0000 mgr.smithi084.itrhok (mgr.14156) 5 : cephadm [INF] Deploying daemon alertmanager.smithi084 on smithi084 2024-01-13T09:46:06.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:05 smithi084 conmon[33408]: audit 2024-01-13T09:46:04.745852+0000 mgr.smithi084.itrhok (mgr.14156) 6 : audit [DBG] from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T09:46:06.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:05 smithi084 conmon[33408]: audit 2024-01-13T09:46:04.748237+0000 mon.smithi084 (mon.0) 98 : audit [INF] from='mgr.14156 172.21.15.84:0/355417686' entity='mgr.smithi084.itrhok' 2024-01-13T09:46:11.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:11 smithi084 conmon[33408]: cluster 2024-01-13T09:46:10.196003+0000 2024-01-13T09:46:11.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:11 smithi084 conmon[33408]: mgr.smithi084.itrhok (mgr.14156) 7 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:46:12.736 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:12 smithi084 conmon[33408]: audit 2024-01-13T09:46:11.376083+0000 mon.smithi084 (mon.0) 99 2024-01-13T09:46:12.736 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:12 smithi084 conmon[33408]: : audit [INF] from='mgr.14156 172.21.15.84:0/355417686' entity='mgr.smithi084.itrhok' 2024-01-13T09:46:12.736 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:12 smithi084 conmon[33408]: audit 2024-01-13T09:46:11.377784+0000 mon.smithi084 (mon.0) 100 : audit [INF] from='mgr.14156 172.21.15.84:0/355417686' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi084", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-13T09:46:12.737 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:12 smithi084 conmon[33408]: audit 2024-01-13T09:46:11.382266+0000 mon.smithi084 (mon.0) 101 : audit [INF] from='mgr.14156 172.21.15.84:0/355417686' entity='mgr.smithi084.itrhok' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi084", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-01-13T09:46:12.737 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:12 smithi084 conmon[33408]: audit 2024-01-13T09:46:11.382835+0000 mon.smithi084 (mon.0) 102 : audit [DBG] from='mgr.14156 172.21.15.84:0/355417686' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:46:12.737 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:12 smithi084 conmon[33408]: cephadm 2024-01-13T09:46:11.383327+0000 mgr.smithi084.itrhok (mgr.14156) 8 : cephadm [INF] Deploying daemon crash.smithi084 on smithi084 2024-01-13T09:46:13.570 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:13 smithi084 conmon[33408]: cluster 2024-01-13T09:46:12.196402+0000 mgr.smithi084.itrhok (mgr. 2024-01-13T09:46:13.570 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:13 smithi084 conmon[33408]: 14156) 9 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:46:15.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:14 smithi084 conmon[33408]: audit 2024-01-13T09:46:13.625727+0000 mon.smithi084 (mon.0) 103 : audit [INF] from='mgr.14156 172.21.15.84:0/355417686' entity='mgr.smithi084.itrhok' 2024-01-13T09:46:15.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:14 smithi084 conmon[33408]: audit 2024-01-13T09:46:13.743741+0000 mon.smithi084 (mon.0) 104 : audit [INF] from='mgr.14156 172.21.15.84:0/355417686' entity='mgr.smithi084.itrhok' 2024-01-13T09:46:15.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:14 smithi084 conmon[33408]: audit 2024-01-13T09:46:13.745506+0000 mon.smithi084 (mon.0) 105 : audit [INF] from='mgr.14156 172.21.15.84:0/355417686' entity='mgr.smithi084.itrhok' 2024-01-13T09:46:15.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:14 smithi084 conmon[33408]: audit 2024-01-13T09:46:13.747378+0000 mon.smithi084 (mon.0) 106 : audit [INF] from='mgr.14156 172.21.15.84:0/355417686' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-13T09:46:15.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:14 smithi084 conmon[33408]: audit 2024-01-13T09:46:13.747696+0000 mgr.smithi084.itrhok (mgr.14156) 10 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-13T09:46:15.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:14 smithi084 conmon[33408]: audit 2024-01-13T09:46:13.749959+0000 mon.smithi084 (mon.0) 107 : audit [INF] from='mgr.14156 172.21.15.84:0/355417686' entity='mgr.smithi084.itrhok' 2024-01-13T09:46:15.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:14 smithi084 conmon[33408]: cephadm 2024-01-13T09:46:13.753115+0000 mgr.smithi084.itrhok (mgr.14156) 11 : cephadm [INF] Deploying daemon grafana.smithi084 on smithi084 2024-01-13T09:46:16.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:15 smithi084 conmon[33408]: cluster 2024-01-13T09:46:14.196718 2024-01-13T09:46:16.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:15 smithi084 conmon[33408]: +0000 mgr.smithi084.itrhok (mgr.14156) 12 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:46:16.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:15 smithi084 conmon[33408]: audit 2024-01-13T09:46:15.333689+0000 mon.smithi084 (mon.0) 108 : 2024-01-13T09:46:16.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:15 smithi084 conmon[33408]: audit [INF] from='mgr.14156 172.21.15.84:0/355417686' entity='mgr.smithi084.itrhok' 2024-01-13T09:46:18.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:17 smithi084 conmon[33408]: cluster 2024-01-13T09:46:16.197093+0000 mgr.smithi084.itrhok (mgr.14156) 13 2024-01-13T09:46:18.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:17 smithi084 conmon[33408]: : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:46:20.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:19 smithi084 conmon[33408]: cluster 2024-01-13T09:46:18.197325+0000 2024-01-13T09:46:20.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:19 smithi084 conmon[33408]: mgr.smithi084.itrhok (mgr.14156) 14 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:46:22.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:21 smithi084 conmon[33408]: cluster 2024-01-13T09:46:20. 2024-01-13T09:46:22.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:21 smithi084 conmon[33408]: 197560+0000 mgr.smithi084.itrhok (mgr.14156) 15 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:46:24.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:23 smithi084 conmon[33408]: cluster 2024-01-13T09:46:22.197783+0000 mgr.smithi084.itrhok (mgr.14156) 16 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:46:26.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:25 smithi084 conmon[33408]: cluster 2024-01-13T09:46:24.198085+0000 mgr.smithi084.itrhok (mgr.14156) 17 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:46:28.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:27 smithi084 conmon[33408]: cluster 2024-01-13T09:46:26.198418+0000 mgr.smithi084.itrhok (mgr.14156) 18 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:46:30.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:29 smithi084 conmon[33408]: cluster 2024-01-13T09:46:28.198823+0000 mgr.smithi084.itrhok (mgr.14156) 19 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:46:32.004 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:31 smithi084 conmon[33408]: cluster 2024-01-13T09:46:30.199201+0000 mgr.smithi084.itrhok (mgr 2024-01-13T09:46:32.004 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:31 smithi084 conmon[33408]: .14156) 20 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:46:33.053 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:32 smithi084 conmon[33408]: audit 2024-01-13T09:46:31.976530+0000 mon.smithi084 (mon.0) 109 2024-01-13T09:46:33.054 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:32 smithi084 conmon[33408]: : audit [INF] from='mgr.14156 172.21.15.84:0/355417686' entity='mgr.smithi084.itrhok' 2024-01-13T09:46:33.054 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:32 smithi084 conmon[33408]: cephadm 2024-01-13T09:46:31.978548+0000 mgr.smithi084.itrhok (mgr.14156) 21 : cephadm [INF] Deploying daemon node-exporter.smithi084 on smithi084 2024-01-13T09:46:33.054 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:32 smithi084 conmon[33408]: cluster 2024-01-13T09:46:32.199497+0000 mgr.smithi084.itrhok (mgr.14156) 22 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:46:35.302 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:35 smithi084 conmon[33408]: cluster 2024-01-13T09:46:34.199769+0000 mgr.smithi084.itrhok (mgr.14156) 23 : 2024-01-13T09:46:35.302 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:35 smithi084 conmon[33408]: cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:46:36.434 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:36 smithi084 conmon[33408]: audit 2024-01-13T09:46:35.338269+0000 mon.smithi084 (mon. 2024-01-13T09:46:36.434 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:36 smithi084 conmon[33408]: 0) 110 : audit [INF] from='mgr.14156 172.21.15.84:0/355417686' entity='mgr.smithi084.itrhok' 2024-01-13T09:46:36.434 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:36 smithi084 conmon[33408]: audit 2024-01-13T09:46:35.656973+0000 mon.smithi084 (mon.0) 2024-01-13T09:46:36.434 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:36 smithi084 conmon[33408]: 111 : audit [INF] from='mgr.14156 172.21.15.84:0/355417686' entity='mgr.smithi084.itrhok' 2024-01-13T09:46:36.434 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:36 smithi084 conmon[33408]: audit 2024-01-13T09: 2024-01-13T09:46:36.434 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:36 smithi084 conmon[33408]: 46:35.659055+0000 mon.smithi084 (mon 2024-01-13T09:46:36.434 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:36 smithi084 conmon[33408]: .0) 112 : audit [INF] from='mgr.14156 172.21.15.84:0/355417686' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-01-13T09:46:36.434 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:36 smithi084 conmon[33408]: 2024-01-13T09:46:37.759 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:37 smithi084 conmon[33408]: audit 2024-01-13T09:46:36.659895+0000 mon.smithi084 (mon.0) 113 : 2024-01-13T09:46:37.759 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:37 smithi084 conmon[33408]: audit [INF] from='mgr.14156 172.21.15.84:0/355417686' entity='mgr.smithi084.itrhok' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-01-13T09:46:37.759 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:37 smithi084 conmon[33408]: cluster 2024-01-13T09:46:36.659948+0000 mon.smithi084 (mon.0) 114 : cluster [DBG] mgrmap e14: smithi084.itrhok(active, since 46s) 2024-01-13T09:46:42.255 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: cluster 2024-01-13T09:46:42.055505+0000 mon.smithi084 (mon.0 2024-01-13T09:46:42.255 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: ) 115 : cluster [INF] Active manager daemon smithi084.itrhok restarted 2024-01-13T09:46:42.255 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: cluster 2024-01-13T09:46: 2024-01-13T09:46:42.255 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: 42.055765+0000 mon.smithi084 (mon.0) 116 : cluster [INF] 2024-01-13T09:46:42.255 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: Activating manager daemon smithi084.itrhok 2024-01-13T09:46:42.255 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: cluster 2024- 2024-01-13T09:46:42.255 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: 01-13T09:46:42. 2024-01-13T09:46:42.255 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: 058519+0000 mon.smithi084 (mon.0) 2024-01-13T09:46:42.255 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: 117 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-01-13T09:46:42.256 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: cluster 2024-01-13T09:46:42.256 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: 2024-01-13T09:46 2024-01-13T09:46:42.256 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: :42.058626+0000 mon.smithi084 (mon 2024-01-13T09:46:42.256 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: .0) 118 : cluster [DBG] mgrmap e15: smithi084.itrhok(active, starting, since 0.00295066s) 2024-01-13T09:46:42.256 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: 2024-01-13T09:46:42.256 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: audit 2024-01- 2024-01-13T09:46:42.256 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: 13T09:46:42.059608+0000 2024-01-13T09:46:42.256 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: mon.smithi084 (mon.0) 119 : 2024-01-13T09:46:42.256 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mon metadata", "id": "smithi084"}]: dispatch 2024-01-13T09:46:42.256 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: audit 2024 2024-01-13T09:46:42.256 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: -01-13T09:46:42 2024-01-13T09:46:42.256 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: .060164+0000 mon.smithi084 (mon.0 2024-01-13T09:46:42.256 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: ) 120 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mgr metadata", "who": "smithi084.itrhok", "id": "smithi084.itrhok"}]: dispatch 2024-01-13T09:46:42.257 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: audit 2024- 2024-01-13T09:46:42.257 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: 01-13T09:46:42.060835 2024-01-13T09:46:42.257 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: +0000 mon.smithi084 (mon.0) 121 2024-01-13T09:46:42.257 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-13T09:46:42.257 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: audit 2024-01-13T09:46:42.257 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: 2024-01-13T09:46 2024-01-13T09:46:42.257 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: :42.061066+0000 mon.smithi084 (mon 2024-01-13T09:46:42.257 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: .0) 122 : audit [DBG] 2024-01-13T09:46:42.257 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-13T09:46:42.257 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: audit 2024-01 2024-01-13T09:46:42.257 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: -13T09:46:42. 2024-01-13T09:46:42.257 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: 061270+0000 mon.smithi084 (mon.0) 2024-01-13T09:46:42.257 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: 123 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-13T09:46:42.257 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: cluster 2024 2024-01-13T09:46:42.257 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: -01-13T09:46:42 2024-01-13T09:46:42.258 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: .075021+0000 mon.smithi084 (mon.0 2024-01-13T09:46:42.258 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:42 smithi084 conmon[33408]: ) 124 : cluster [INF] Manager daemon smithi084.itrhok is now available 2024-01-13T09:46:43.361 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:43 smithi084 conmon[33408]: audit 2024-01-13T09:46:42.172137+0000 mon.smithi084 (mon.0 2024-01-13T09:46:43.362 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:43 smithi084 conmon[33408]: ) 125 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T09:46:43.362 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:43 smithi084 conmon[33408]: audit 2024-01-13T09:46:42.173560+0000 mon.smithi084 (mon.0) 126 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T09:46:43.362 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:43 smithi084 conmon[33408]: 2024-01-13T09:46:43.362 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:43 smithi084 conmon[33408]: audit 2024-01-13T09 2024-01-13T09:46:43.362 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:43 smithi084 conmon[33408]: :46:42.196497+0000 mon.smithi084 (mon.0) 127 : audit 2024-01-13T09:46:43.362 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:43 smithi084 conmon[33408]: [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:46:43.362 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:43 smithi084 conmon[33408]: audit 2024- 2024-01-13T09:46:43.362 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:43 smithi084 conmon[33408]: 01-13T09:46:42. 2024-01-13T09:46:43.362 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:43 smithi084 conmon[33408]: 206794+0000 mon.smithi084 (mon.0) 128 2024-01-13T09:46:43.362 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:43 smithi084 conmon[33408]: : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.itrhok/mirror_snapshot_schedule"}]: dispatch 2024-01-13T09:46:43.362 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:43 smithi084 conmon[33408]: audit 2024-01-13T09:46:43.362 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:43 smithi084 conmon[33408]: 2024-01-13T09:46:42.211455+0000 2024-01-13T09:46:43.362 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:43 smithi084 conmon[33408]: mon.smithi084 (mon.0) 129 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.itrhok/trash_purge_schedule"}]: dispatch 2024-01-13T09:46:43.363 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:43 smithi084 conmon[33408]: 2024-01-13T09:46:43.363 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:43 smithi084 conmon[33408]: cluster 2024- 2024-01-13T09:46:43.363 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:43 smithi084 conmon[33408]: 01-13T09:46:43.060502+0000 mon.smithi084 (mon.0 2024-01-13T09:46:43.363 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:43 smithi084 conmon[33408]: ) 130 : cluster [DBG] mgrmap e16: smithi084.itrhok(active, since 1.00482s) 2024-01-13T09:46:45.248 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:45 smithi084 conmon[33408]: cluster 2024-01-13T09:46:44.110836+0000 mon.smithi084 (mon.0) 131 : cluster [DBG] mgrmap e17: smithi084.itrhok(active, since 2s) 2024-01-13T09:46:46.312 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:46 smithi084 conmon[33408]: audit 2024-01-13T09:46:45.287181+0000 mon.smithi084 (mon.0) 132 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:46:46.313 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:46 smithi084 conmon[33408]: audit 2024-01-13T09:46:45.590014 2024-01-13T09:46:46.313 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:46 smithi084 conmon[33408]: +0000 mon.smithi084 (mon.0) 133 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config rm", "who": "osd/host:smithi084", "name": "osd_memory_target"}]: dispatch 2024-01-13T09:46:46.313 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:46 smithi084 conmon[33408]: audit 2024-01-13T09:46:45.590776+0000 mon.smithi084 (mon.0) 134 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:46:46.313 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:46 smithi084 conmon[33408]: audit 2024-01-13T09:46:45.591274+0000 mon.smithi084 (mon.0) 135 2024-01-13T09:46:46.313 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:46 smithi084 conmon[33408]: : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T09:46:46.313 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:46 smithi084 conmon[33408]: cephadm 2024-01-13T09:46: 2024-01-13T09:46:46.313 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:46 smithi084 conmon[33408]: 45.592228+0000 mgr.smithi084.itrhok (mgr.14180) 1 : cephadm 2024-01-13T09:46:46.313 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:46 smithi084 conmon[33408]: [INF] Updating smithi084:/etc/ceph/ceph.conf 2024-01-13T09:46:46.314 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:46 smithi084 conmon[33408]: cephadm 2024-01-13T09:46:46.314 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:46 smithi084 conmon[33408]: 2024-01-13T09:46:45.617827+0000 mgr.smithi084.itrhok (mgr 2024-01-13T09:46:46.314 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:46 smithi084 conmon[33408]: .14180) 2 : cephadm [INF] Updating smithi084:/etc/ceph/ceph.client.admin.keyring 2024-01-13T09:46:46.314 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:46 smithi084 conmon[33408]: audit 2024-01-13T09 2024-01-13T09:46:46.314 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:46 smithi084 conmon[33408]: :46:45.696597+0000 mon.smithi084 (mon.0) 2024-01-13T09:46:46.314 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:46 smithi084 conmon[33408]: 136 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:46:46.314 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:46 smithi084 conmon[33408]: audit 2024-01-13T09:46: 2024-01-13T09:46:46.314 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:46 smithi084 conmon[33408]: 45.699006+0000 mon.smithi084 (mon.0) 137 : audit 2024-01-13T09:46:46.315 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:46 smithi084 conmon[33408]: [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:46:47.419 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:47 smithi084 conmon[33408]: cephadm 2024-01-13T09:46:45.710133+0000 mgr.smithi084.itrhok (mgr.14180) 3 : cephadm [INF] Deploying daemon prometheus.smithi084 on smithi084 2024-01-13T09:46:47.911 INFO:teuthology.orchestra.run.smithi102.stdout:Added host 'smithi102' with addr '172.21.15.102' 2024-01-13T09:46:48.497 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph orch host ls --format=json 2024-01-13T09:46:48.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:48 smithi084 conmon[33408]: audit 2024-01-13T09:46:47.297273+0000 2024-01-13T09:46:48.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:48 smithi084 conmon[33408]: mgr.smithi084.itrhok (mgr.14180) 4 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi102", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T09:46:48.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:48 smithi084 conmon[33408]: audit 2024-01-13T09: 2024-01-13T09:46:48.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:48 smithi084 conmon[33408]: 46:47.919928+0000 mon.smithi084 (mon.0) 138 : audit 2024-01-13T09:46:48.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:48 smithi084 conmon[33408]: [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:46:49.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:49 smithi084 conmon[33408]: cephadm 2024-01-13T09:46:47.920228+0000 mgr.smithi084.itrhok (mgr.14180 2024-01-13T09:46:49.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:49 smithi084 conmon[33408]: ) 5 : cephadm [INF] Added host smithi102 2024-01-13T09:46:50.196 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-01-13T09:46:50.196 INFO:teuthology.orchestra.run.smithi102.stdout:[{"addr": "172.21.15.84", "hostname": "smithi084", "labels": [], "status": ""}, {"addr": "172.21.15.102", "hostname": "smithi102", "labels": [], "status": ""}] 2024-01-13T09:46:50.728 INFO:tasks.cephadm:Setting crush tunables to default 2024-01-13T09:46:50.728 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd crush tunables default 2024-01-13T09:46:51.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:51 smithi084 conmon[33408]: audit 2024-01-13T09:46:50.205518 2024-01-13T09:46:51.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:51 smithi084 conmon[33408]: +0000 mgr.smithi084.itrhok (mgr.14180) 6 : audit [DBG] from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-13T09:46:58.648 INFO:teuthology.orchestra.run.smithi084.stderr:adjusted tunables profile to default 2024-01-13T09:46:58.811 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:58 smithi084 conmon[33408]: audit 2024-01-13T09:46:58.601376+0000 mon.smithi084 (mon.0) 139 : audit [INF] from='client.? 172.21.15.84:0/1728316719' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-01-13T09:46:59.255 INFO:tasks.cephadm:Adding mon.smithi084 on smithi084 2024-01-13T09:46:59.255 INFO:tasks.cephadm:Adding mon.smithi102 on smithi102 2024-01-13T09:46:59.255 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph orch apply mon '2;smithi084:172.21.15.84=smithi084;smithi102:172.21.15.102=smithi102' 2024-01-13T09:46:59.771 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:59 smithi084 conmon[33408]: audit 2024-01-13T09:46:58.654858+0000 mon.smithi084 (mon.0) 140 : audit [INF] from='client.? 172.21.15.84:0/1728316719' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-01-13T09:46:59.771 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:46:59 smithi084 conmon[33408]: cluster 2024-01-13T09:46:58.654942+0000 mon.smithi084 (mon.0) 141 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-01-13T09:47:00.948 INFO:teuthology.orchestra.run.smithi102.stdout:Scheduled mon update... 2024-01-13T09:47:01.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:00 smithi084 conmon[33408]: audit 2024-01-13T09:46:59.748687+0000 mon.smithi084 (mon.0) 2024-01-13T09:47:01.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:00 smithi084 conmon[33408]: 142 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:01.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:00 smithi084 conmon[33408]: audit 2024-01-13T09:46:59.749813+0000 mon.smithi084 (mon.0) 143 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T09:47:01.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:00 smithi084 conmon[33408]: audit 2024-01-13T09:47:00.213487+0000 mon.smithi084 (mon.0) 144 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:01.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:00 smithi084 conmon[33408]: audit 2024-01-13T09:47:00.735252+0000 mon.smithi084 (mon.0) 145 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:01.514 DEBUG:teuthology.orchestra.run.smithi102:mon.smithi102> sudo journalctl -f -n 0 -u ceph-503cf380-b1f8-11ee-95ac-87774f69a715@mon.smithi102.service 2024-01-13T09:47:01.516 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T09:47:01.517 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T09:47:01.546 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:-- Logs begin at Sat 2024-01-13 09:33:35 UTC. -- 2024-01-13T09:47:02.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:01 smithi084 conmon[33408]: audit 2024-01-13T09:47:00.954101+0000 mgr.smithi084.itrhok (mgr.14180) 7 : audit 2024-01-13T09:47:02.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:01 smithi084 conmon[33408]: [DBG] from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi084:172.21.15.84=smithi084;smithi102:172.21.15.102=smithi102", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T09:47:02.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:01 smithi084 conmon[33408]: cephadm 2024-01-13T09:47:00.956109+0000 mgr.smithi084.itrhok (mgr.14180) 8 : cephadm [INF] Saving service mon spec with placement smithi084:172.21.15.84=smithi084;smithi102:172.21.15.102=smithi102;count:2 2024-01-13T09:47:02.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:01 smithi084 conmon[33408]: audit 2024-01-13T09:47:00.958458+0000 mon.smithi084 2024-01-13T09:47:02.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:01 smithi084 conmon[33408]: (mon.0) 146 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:03.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:03 smithi084 conmon[33408]: cluster 2024-01-13T09:47:02.062308+0000 mgr.smithi084.itrhok (mgr.14180) 9 : cluster [DBG] 2024-01-13T09:47:03.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:03 smithi084 conmon[33408]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:03.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:03 smithi084 conmon[33408]: audit 2024-01-13T09:47:02.200112+0000 mon.smithi084 (mon.0) 147 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:03.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:03 smithi084 conmon[33408]: audit 2024-01-13T09:47:02.809711+0000 mon.smithi084 (mon.0) 148 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:04.063 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-01-13T09:47:04.063 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":1,"fsid":"503cf380-b1f8-11ee-95ac-87774f69a715","modified":"2024-01-13T09:44:57.585744Z","created":"2024-01-13T09:44:57.585744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T09:47:04.065 INFO:teuthology.orchestra.run.smithi102.stderr:dumped monmap epoch 1 2024-01-13T09:47:04.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:04 smithi084 conmon[33408]: audit 2024-01-13T09:47:04.072790+0000 mon.smithi084 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.102:0/3614394332' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T09:47:05.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:05 smithi084 conmon[33408]: cluster 2024-01-13T09:47:04.062733+0000 mgr.smithi084.itrhok (mgr.14180) 10 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:05.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:05 smithi084 conmon[33408]: audit 2024-01-13T09:47:05.193951+0000 mon.smithi084 (mon.0) 150 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:05.910 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T09:47:05.910 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T09:47:07.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:06 smithi084 conmon[33408]: audit 2024-01-13T09:47:05.628408+0000 mon.smithi084 (mon.0) 151 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:07.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:06 smithi084 conmon[33408]: 2024-01-13T09:47:07.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:06 smithi084 conmon[33408]: audit 2024-01-13T09:47:05.629251+0000 mon.smithi084 (mon.0) 152 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config rm", "who": "osd/host:smithi102", "name": "osd_memory_target"}]: dispatch 2024-01-13T09:47:07.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:06 smithi084 conmon[33408]: audit 2024-01-13T09:47:05.630238+0000 mon.smithi084 (mon.0) 153 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:47:07.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:06 smithi084 conmon[33408]: audit 2024-01-13T09:47:05 2024-01-13T09:47:07.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:06 smithi084 conmon[33408]: .631052+0000 mon.smithi084 (mon.0) 154 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T09:47:07.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:06 smithi084 conmon[33408]: cephadm 2024-01-13T09:47:05.632332+0000 mgr.smithi084.itrhok (mgr.14180) 11 : cephadm [INF] Updating smithi102:/etc/ceph/ceph.conf 2024-01-13T09:47:07.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:06 smithi084 conmon[33408]: cephadm 2024-01-13T09:47:05.651472+0000 mgr.smithi084.itrhok (mgr.14180) 12 : cephadm [INF] Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-01-13T09:47:07.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:06 smithi084 conmon[33408]: audit 2024-01-13T09:47:05.670898+0000 mon.smithi084 (mon.0) 155 : audit 2024-01-13T09:47:07.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:06 smithi084 conmon[33408]: [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:07.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:06 smithi084 conmon[33408]: audit 2024-01-13T09:47:05.674245+0000 2024-01-13T09:47:07.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:06 smithi084 conmon[33408]: mon.smithi084 (mon.0) 156 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:07.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:06 smithi084 conmon[33408]: audit 2024-01-13T09: 2024-01-13T09:47:07.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:06 smithi084 conmon[33408]: 47:05.675874+0000 mon.smithi084 (mon.0) 157 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi102", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-13T09:47:07.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:06 smithi084 conmon[33408]: audit 2024-01-13T09 2024-01-13T09:47:07.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:06 smithi084 conmon[33408]: :47:05.678434+0000 mon.smithi084 (mon.0) 158 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi102", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-01-13T09:47:07.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:06 smithi084 conmon[33408]: audit 2024-01-13 2024-01-13T09:47:07.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:06 smithi084 conmon[33408]: T09:47:05.679201+0000 mon.smithi084 (mon.0) 159 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:47:07.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:06 smithi084 conmon[33408]: cephadm 2024-01-13 2024-01-13T09:47:07.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:06 smithi084 conmon[33408]: T09:47:05.679880+0000 mgr.smithi084.itrhok (mgr.14180) 13 : cephadm [INF] Deploying daemon crash.smithi102 on smithi102 2024-01-13T09:47:08.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:07 smithi084 conmon[33408]: cluster 2024-01-13T09:47:06.062998+0000 mgr.smithi084.itrhok (mgr.14180) 14 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:08.524 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-01-13T09:47:08.524 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":1,"fsid":"503cf380-b1f8-11ee-95ac-87774f69a715","modified":"2024-01-13T09:44:57.585744Z","created":"2024-01-13T09:44:57.585744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T09:47:08.525 INFO:teuthology.orchestra.run.smithi102.stderr:dumped monmap epoch 1 2024-01-13T09:47:09.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:08 smithi084 conmon[33408]: audit 2024-01-13T09:47:08.533219+0000 mon.smithi084 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.102:0/1384237618' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T09:47:10.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:09 smithi084 conmon[33408]: cluster 2024-01-13T09:47:08.063248+0000 mgr.smithi084.itrhok (mgr.14180) 15 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:10.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:09 smithi084 conmon[33408]: audit 2024-01-13T09:47:08.860041+0000 mon.smithi084 (mon.0) 161 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:10.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:09 smithi084 conmon[33408]: audit 2024-01-13T09:47:08.862103+0000 mon.smithi084 (mon.0) 162 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi102.yjovaz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-13T09:47:10.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:09 smithi084 conmon[33408]: audit 2024-01-13T09:47:08.864121+0000 mon.smithi084 (mon.0) 163 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi102.yjovaz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-01-13T09:47:10.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:09 smithi084 conmon[33408]: audit 2024-01-13T09:47:08 2024-01-13T09:47:10.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:09 smithi084 conmon[33408]: .864926+0000 mon.smithi084 (mon.0) 164 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-13T09:47:10.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:09 smithi084 conmon[33408]: audit 2024-01-13T09:47:08.865823+0000 mon.smithi084 (mon.0) 165 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:47:10.412 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T09:47:10.413 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T09:47:11.026 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:10 smithi084 conmon[33408]: cephadm 2024-01-13T09:47:08.866498+0000 mgr.smithi084.itrhok (mgr.14180) 16 : cephadm [INF] Deploying daemon mgr.smithi102.yjovaz on smithi102 2024-01-13T09:47:12.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:11 smithi084 conmon[33408]: cluster 2024-01-13T09:47:10.063623+0000 mgr.smithi084.itrhok (mgr.14180) 17 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:12.561 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-01-13T09:47:12.561 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":1,"fsid":"503cf380-b1f8-11ee-95ac-87774f69a715","modified":"2024-01-13T09:44:57.585744Z","created":"2024-01-13T09:44:57.585744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T09:47:12.563 INFO:teuthology.orchestra.run.smithi102.stderr:dumped monmap epoch 1 2024-01-13T09:47:13.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:12 smithi084 conmon[33408]: audit 2024-01-13T09:47:11.820181+0000 mon.smithi084 (mon.0) 166 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:13.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:12 smithi084 conmon[33408]: audit 2024-01-13T09:47:11.822217+0000 mon.smithi084 (mon.0) 167 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-13T09:47:13.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:12 smithi084 conmon[33408]: audit 2024-01-13T09:47:11.823002+0000 mon.smithi084 (mon.0) 168 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:47:13.278 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:12 smithi084 conmon[33408]: cephadm 2024-01-13T09:47:11.823620+0000 mgr.smithi084.itrhok (mgr.14180) 2024-01-13T09:47:13.278 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:12 smithi084 conmon[33408]: 18 : cephadm [INF] Deploying daemon mon.smithi102 on smithi102 2024-01-13T09:47:13.278 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:12 smithi084 conmon[33408]: cluster 2024-01-13T09:47:12.064024+0000 mgr.smithi084.itrhok (mgr.14180) 19 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:13.278 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:12 smithi084 conmon[33408]: audit 2024- 2024-01-13T09:47:13.278 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:12 smithi084 conmon[33408]: 01-13T09:47:12.206634+0000 mon.smithi084 (mon.0) 169 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:13.278 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:12 smithi084 conmon[33408]: audit 2024-01-13T09:47:12.570607+0000 mon.smithi084 (mon 2024-01-13T09:47:13.278 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:12 smithi084 conmon[33408]: .0) 170 : audit [DBG] from='client.? 172.21.15.102:0/1847665985' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T09:47:14.588 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T09:47:14.588 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T09:47:15.130 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:14 smithi102 systemd[1]: Starting Ceph mon.smithi102 for 503cf380-b1f8-11ee-95ac-87774f69a715... 2024-01-13T09:47:15.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:15 smithi084 conmon[33408]: cluster 2024-01-13T09:47:14.064396+0000 mgr.smithi084.itrhok (mgr.14180) 20 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:15.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:15 smithi084 conmon[33408]: 2024-01-13T09:47:15.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.589+0000 7fc08c961880 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-13T09:47:15.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.589+0000 7fc08c961880 0 ceph version 16.2.14-456-g445562ab (445562ab4bc3ddfb386936119050695810860bcb) pacific (stable), process ceph-mon, pid 7 2024-01-13T09:47:15.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.590+0000 7fc08c961880 0 pidfile_write: ignore empty --pid-file 2024-01-13T09:47:15.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.594+0000 7fc08c961880 0 load: jerasure load: lrc load: isa 2024-01-13T09:47:15.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: RocksDB version: 6.8.1 2024-01-13T09:47:15.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: 2024-01-13T09:47:15.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-13T09:47:15.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Compile date Jan 12 2024 2024-01-13T09:47:15.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: DB SUMMARY 2024-01-13T09:47:15.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: 2024-01-13T09:47:15.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: CURRENT file: CURRENT 2024-01-13T09:47:15.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: 2024-01-13T09:47:15.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: IDENTITY file: IDENTITY 2024-01-13T09:47:15.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: 2024-01-13T09:47:15.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: MANIFEST file: MANIFEST-000001 size: 13 Bytes 2024-01-13T09:47:15.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: 2024-01-13T09:47:15.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi102/store.db dir, Total Num: 0, files: 2024-01-13T09:47:15.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: 2024-01-13T09:47:15.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi102/store.db: 000003.log size: 511 ; 2024-01-13T09:47:15.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: 2024-01-13T09:47:15.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.error_if_exists: 0 2024-01-13T09:47:15.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.create_if_missing: 0 2024-01-13T09:47:15.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.paranoid_checks: 1 2024-01-13T09:47:15.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.env: 0x55e9a0ee6080 2024-01-13T09:47:15.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.fs: Posix File System 2024-01-13T09:47:15.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.info_log: 0x55e9a2c61d20 2024-01-13T09:47:15.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-13T09:47:15.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.statistics: (nil) 2024-01-13T09:47:15.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.use_fsync: 0 2024-01-13T09:47:15.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.max_log_file_size: 0 2024-01-13T09:47:15.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-13T09:47:15.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-13T09:47:15.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-13T09:47:15.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-13T09:47:15.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.allow_fallocate: 1 2024-01-13T09:47:15.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-13T09:47:15.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-13T09:47:15.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.use_direct_reads: 0 2024-01-13T09:47:15.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-13T09:47:15.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.create_missing_column_families: 0 2024-01-13T09:47:15.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.db_log_dir: 2024-01-13T09:47:15.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi102/store.db 2024-01-13T09:47:15.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-13T09:47:15.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.max_subcompactions: 1 2024-01-13T09:47:15.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.max_background_flushes: -1 2024-01-13T09:47:15.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-13T09:47:15.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-13T09:47:15.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-13T09:47:15.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-13T09:47:15.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-13T09:47:15.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.advise_random_on_open: 1 2024-01-13T09:47:15.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-13T09:47:15.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.write_buffer_manager: 0x55e9a38b8480 2024-01-13T09:47:15.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-13T09:47:15.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-13T09:47:15.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-13T09:47:15.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 bash[37481]: 76ea6d70fd53f807234265f8a067c2c0cf2bffc44f99b9b1155de92efca4eb30 2024-01-13T09:47:15.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-13T09:47:15.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.rate_limiter: (nil) 2024-01-13T09:47:15.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-13T09:47:15.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-13T09:47:15.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-13T09:47:15.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-13T09:47:15.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.unordered_write: 0 2024-01-13T09:47:15.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-13T09:47:15.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-13T09:47:15.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-13T09:47:15.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-13T09:47:15.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.row_cache: None 2024-01-13T09:47:15.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.wal_filter: None 2024-01-13T09:47:15.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-13T09:47:15.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-13T09:47:15.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.preserve_deletes: 0 2024-01-13T09:47:15.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.two_write_queues: 0 2024-01-13T09:47:15.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.manual_wal_flush: 0 2024-01-13T09:47:15.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.atomic_flush: 0 2024-01-13T09:47:15.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-13T09:47:15.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-13T09:47:15.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-13T09:47:15.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.log_readahead_size: 0 2024-01-13T09:47:15.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-13T09:47:15.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.max_background_jobs: 2 2024-01-13T09:47:15.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.max_background_compactions: -1 2024-01-13T09:47:15.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-13T09:47:15.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-13T09:47:15.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-13T09:47:15.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.max_total_wal_size: 0 2024-01-13T09:47:15.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-13T09:47:15.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-13T09:47:15.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-13T09:47:15.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-13T09:47:15.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.max_open_files: -1 2024-01-13T09:47:15.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.bytes_per_sync: 0 2024-01-13T09:47:15.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-13T09:47:15.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-13T09:47:15.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-13T09:47:15.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Compression algorithms supported: 2024-01-13T09:47:15.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-13T09:47:15.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: kZSTD supported: 0 2024-01-13T09:47:15.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: kXpressCompression supported: 0 2024-01-13T09:47:15.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-13T09:47:15.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: kLZ4Compression supported: 1 2024-01-13T09:47:15.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: kBZip2Compression supported: 0 2024-01-13T09:47:15.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: kZlibCompression supported: 1 2024-01-13T09:47:15.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: kSnappyCompression supported: 1 2024-01-13T09:47:15.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-13T09:47:15.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi102/store.db/MANIFEST-000001 2024-01-13T09:47:15.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: 2024-01-13T09:47:15.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-13T09:47:15.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: 2024-01-13T09:47:15.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-13T09:47:15.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.merge_operator: 2024-01-13T09:47:15.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.compaction_filter: None 2024-01-13T09:47:15.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.compaction_filter_factory: None 2024-01-13T09:47:15.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-13T09:47:15.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-13T09:47:15.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e9a2c0bd28) 2024-01-13T09:47:15.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: cache_index_and_filter_blocks: 1 2024-01-13T09:47:15.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-13T09:47:15.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-13T09:47:15.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: pin_top_level_index_and_filter: 1 2024-01-13T09:47:15.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: index_type: 0 2024-01-13T09:47:15.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: data_block_index_type: 0 2024-01-13T09:47:15.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: index_shortening: 1 2024-01-13T09:47:15.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: data_block_hash_table_util_ratio: 0.750000 2024-01-13T09:47:15.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: hash_index_allow_collision: 1 2024-01-13T09:47:15.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: checksum: 1 2024-01-13T09:47:15.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: no_block_cache: 0 2024-01-13T09:47:15.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: block_cache: 0x55e9a2c44f10 2024-01-13T09:47:15.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: block_cache_name: BinnedLRUCache 2024-01-13T09:47:15.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: block_cache_options: 2024-01-13T09:47:15.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: capacity : 536870912 2024-01-13T09:47:15.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: num_shard_bits : 4 2024-01-13T09:47:15.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: strict_capacity_limit : 0 2024-01-13T09:47:15.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: high_pri_pool_ratio: 0.000 2024-01-13T09:47:15.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: block_cache_compressed: (nil) 2024-01-13T09:47:15.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: persistent_cache: (nil) 2024-01-13T09:47:15.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: block_size: 4096 2024-01-13T09:47:15.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: block_size_deviation: 10 2024-01-13T09:47:15.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: block_restart_interval: 16 2024-01-13T09:47:15.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: index_block_restart_interval: 1 2024-01-13T09:47:15.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: metadata_block_size: 4096 2024-01-13T09:47:15.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: partition_filters: 0 2024-01-13T09:47:15.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: use_delta_encoding: 1 2024-01-13T09:47:15.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-13T09:47:15.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: whole_key_filtering: 1 2024-01-13T09:47:15.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: verify_compression: 0 2024-01-13T09:47:15.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: read_amp_bytes_per_bit: 0 2024-01-13T09:47:15.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: format_version: 2 2024-01-13T09:47:15.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: enable_index_compression: 1 2024-01-13T09:47:15.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: block_align: 0 2024-01-13T09:47:15.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: 2024-01-13T09:47:15.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-13T09:47:15.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-13T09:47:15.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.compression: NoCompression 2024-01-13T09:47:15.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-13T09:47:15.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-13T09:47:15.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-13T09:47:15.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.num_levels: 7 2024-01-13T09:47:15.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-13T09:47:15.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-13T09:47:15.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-13T09:47:15.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-13T09:47:15.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-13T09:47:15.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-13T09:47:15.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-13T09:47:15.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-13T09:47:15.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-13T09:47:15.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-13T09:47:15.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.compression_opts.level: 32767 2024-01-13T09:47:15.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-13T09:47:15.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-13T09:47:15.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-13T09:47:15.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.compression_opts.enabled: false 2024-01-13T09:47:15.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-13T09:47:15.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-13T09:47:15.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-13T09:47:15.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-13T09:47:15.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-13T09:47:15.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-13T09:47:15.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-13T09:47:15.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-13T09:47:15.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-13T09:47:15.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-13T09:47:15.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-13T09:47:15.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-13T09:47:15.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-13T09:47:15.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-13T09:47:15.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-13T09:47:15.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-13T09:47:15.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-13T09:47:15.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.arena_block_size: 4194304 2024-01-13T09:47:15.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-13T09:47:15.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-13T09:47:15.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-13T09:47:15.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-13T09:47:15.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-13T09:47:15.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-13T09:47:15.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-13T09:47:15.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-13T09:47:15.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-13T09:47:15.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-13T09:47:15.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-13T09:47:15.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-13T09:47:15.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.compaction_option 2024-01-13T09:47:15.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: s_fifo.max_table_files_size: 1073741824 2024-01-13T09:47:15.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-13T09:47:15.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.table_properties_collectors: 2024-01-13T09:47:15.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.inplace_update_support: 0 2024-01-13T09:47:15.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-13T09:47:15.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-13T09:47:15.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-13T09:47:15.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-13T09:47:15.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.bloom_locality: 0 2024-01-13T09:47:15.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.max_successive_merges: 0 2024-01-13T09:47:15.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-13T09:47:15.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-13T09:47:15.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.force_consistency_checks: 0 2024-01-13T09:47:15.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-13T09:47:15.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.ttl: 2592000 2024-01-13T09:47:15.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.595+0000 7fc08c961880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-13T09:47:15.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.596+0000 7fc08c961880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi102/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-13T09:47:15.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: 2024-01-13T09:47:15.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.596+0000 7fc08c961880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-01-13T09:47:15.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: 2024-01-13T09:47:15.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.596+0000 7fc08c961880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705139235597087, "job": 1, "event": "recovery_started", "log_files": [3]} 2024-01-13T09:47:15.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.596+0000 7fc08c961880 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #3 mode 2 2024-01-13T09:47:15.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.596+0000 7fc08c961880 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-01-13T09:47:15.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.596+0000 7fc08c961880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705139235597604, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 4, "file_size": 1437, "table_properties": {"data_size": 523, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1705139235, "oldest_key_time": 3, "file_creation_time": 0}} 2024-01-13T09:47:15.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.596+0000 7fc08c961880 4 rocksdb: [version_set.cc:3826] Creating manifest 5 2024-01-13T09:47:15.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: 2024-01-13T09:47:15.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.597+0000 7fc08c961880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705139235598086, "job": 1, "event": "recovery_finished"} 2024-01-13T09:47:15.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.598+0000 7fc08c961880 4 rocksdb: DB pointer 0x55e9a2d0f800 2024-01-13T09:47:15.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.598+0000 7fc073755700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-13T09:47:15.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.598+0000 7fc073755700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-13T09:47:15.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: ** DB Stats ** 2024-01-13T09:47:15.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T09:47:15.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-13T09:47:15.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-13T09:47:15.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-13T09:47:15.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-13T09:47:15.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-13T09:47:15.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-13T09:47:15.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: 2024-01-13T09:47:15.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: ** Compaction Stats [default] ** 2024-01-13T09:47:15.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T09:47:15.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T09:47:15.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.000 0 0 2024-01-13T09:47:15.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.000 0 0 2024-01-13T09:47:15.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.000 0 0 2024-01-13T09:47:15.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: 2024-01-13T09:47:15.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: ** Compaction Stats [default] ** 2024-01-13T09:47:15.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T09:47:15.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T09:47:15.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.9 0.00 0.00 1 0.000 0 0 2024-01-13T09:47:15.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T09:47:15.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-13T09:47:15.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-13T09:47:15.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: AddFile(Total Files): cumulative 0, interval 0 2024-01-13T09:47:15.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-13T09:47:15.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: AddFile(Keys): cumulative 0, interval 0 2024-01-13T09:47:15.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: Cumulative compaction: 0.00 GB write, 0.53 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T09:47:15.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: Interval compaction: 0.00 GB write, 0.53 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T09:47:15.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-13T09:47:15.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: 2024-01-13T09:47:15.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: ** File Read Latency Histogram By Level [default] ** 2024-01-13T09:47:15.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: 2024-01-13T09:47:15.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: ** Compaction Stats [default] ** 2024-01-13T09:47:15.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T09:47:15.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T09:47:15.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.000 0 0 2024-01-13T09:47:15.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.000 0 0 2024-01-13T09:47:15.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-13T09:47:15.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: 2024-01-13T09:47:15.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: ** Compaction Stats [default] ** 2024-01-13T09:47:15.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T09:47:15.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T09:47:15.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.9 0.00 0.00 1 0.000 0 0 2024-01-13T09:47:15.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T09:47:15.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-13T09:47:15.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-13T09:47:15.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: AddFile(Total Files): cumulative 0, interval 0 2024-01-13T09:47:15.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-13T09:47:15.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: AddFile(Keys): cumulative 0, interval 0 2024-01-13T09:47:15.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: Cumulative compaction: 0.00 GB write, 0.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T09:47:15.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T09:47:15.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-13T09:47:15.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: 2024-01-13T09:47:15.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: ** File Read Latency Histogram By Level [default] ** 2024-01-13T09:47:15.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: 2024-01-13T09:47:15.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.598+0000 7fc08c961880 0 mon.smithi102 does not exist in monmap, will attempt to join an existing cluster 2024-01-13T09:47:15.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.598+0000 7fc08c961880 0 using public_addr v2:172.21.15.102:0/0 -> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] 2024-01-13T09:47:15.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.599+0000 7fc08c961880 0 starting mon.smithi102 rank -1 at public addrs [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] at bind addrs [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi102 fsid 503cf380-b1f8-11ee-95ac-87774f69a715 2024-01-13T09:47:15.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.600+0000 7fc08c961880 1 mon.smithi102@-1(???) e0 preinit fsid 503cf380-b1f8-11ee-95ac-87774f69a715 2024-01-13T09:47:15.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 systemd[1]: Started Ceph mon.smithi102 for 503cf380-b1f8-11ee-95ac-87774f69a715. 2024-01-13T09:47:15.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.634+0000 7fc075f5a700 0 mon.smithi102@-1(synchronizing).mds e1 new map 2024-01-13T09:47:15.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.634+0000 7fc075f5a700 0 mon.smithi102@-1(synchronizing).mds e1 print_map 2024-01-13T09:47:15.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: e1 2024-01-13T09:47:15.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: enable_multiple, ever_enabled_multiple: 1,1 2024-01-13T09:47:15.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-13T09:47:15.756 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: legacy client fscid: -1 2024-01-13T09:47:15.756 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: 2024-01-13T09:47:15.756 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: No filesystems configured 2024-01-13T09:47:15.756 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: 2024-01-13T09:47:15.756 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.634+0000 7fc075f5a700 1 mon.smithi102@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-01-13T09:47:15.756 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.634+0000 7fc075f5a700 1 mon.smithi102@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-01-13T09:47:15.756 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.634+0000 7fc075f5a700 1 mon.smithi102@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-01-13T09:47:15.756 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.634+0000 7fc075f5a700 1 mon.smithi102@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-01-13T09:47:15.756 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.634+0000 7fc075f5a700 1 mon.smithi102@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-01-13T09:47:15.756 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.635+0000 7fc075f5a700 1 mon.smithi102@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-01-13T09:47:15.756 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.635+0000 7fc075f5a700 1 mon.smithi102@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-01-13T09:47:15.756 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.637+0000 7fc075f5a700 0 mon.smithi102@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-01-13T09:47:15.756 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.637+0000 7fc075f5a700 0 mon.smithi102@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-01-13T09:47:15.756 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.637+0000 7fc075f5a700 0 mon.smithi102@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-01-13T09:47:15.756 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.637+0000 7fc075f5a700 0 mon.smithi102@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-01-13T09:47:15.757 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: cluster 2024-01-13T09:47:15.757 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: 2024-01-13T09: 2024-01-13T09:47:15.757 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: 47:14.064396+0000 mgr.smithi084.itrhok 2024-01-13T09:47:15.757 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: (mgr.14180) 20 : cluster 2024-01-13T09:47:15.757 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:15.757 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:15 smithi102 conmon[37544]: debug 2024-01-13T09:47:15.637+0000 7fc075f5a700 1 mon.smithi102@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-01-13T09:47:21.020 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: audit 2024-01-13T09:47:15.646516+0000 mon.smithi084 (mon.0) 172 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mon metadata", "id": "smithi084"}]: dispatch 2024-01-13T09:47:21.020 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: cluster 2024-01-13T09:47:15.646657+0000 mon.smithi084 (mon.0) 173 2024-01-13T09:47:21.020 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: : cluster [INF] mon.smithi084 calling monitor election 2024-01-13T09:47:21.020 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: audit 2024-01-13T09:47:15.659575+0000 mon.smithi084 (mon.0) 174 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2024-01-13T09:47:21.020 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: cluster 2024-01-13T09:47:16.064802+0000 mgr.smithi084.itrhok (mgr.14180) 21 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:21.020 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: audit 2024-01-13T09:47:16.642618+0000 mon.smithi084 (mon.0) 175 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2024-01-13T09:47:21.020 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: audit 2024-01-13T09:47:17.152227+0000 mon.smithi084 (mon.0) 176 : audit [DBG] from='mgr.? 172.21.15.102:0/1546808686' entity='mgr.smithi102.yjovaz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi102.yjovaz/crt"}]: dispatch 2024-01-13T09:47:21.021 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: audit 2024-01- 2024-01-13T09:47:21.021 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: 13T09:47:17.642674+0000 mon.smithi084 (mon.0) 177 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2024-01-13T09:47:21.021 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: cluster 2024-01-13T09:47:17.644939+0000 mon.smithi102 (mon.1) 1 : cluster [INF] mon.smithi102 calling monitor election 2024-01-13T09:47:21.021 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: cluster 2024-01-13T09 2024-01-13T09:47:21.021 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: :47:18.065170+0000 mgr.smithi084.itrhok (mgr.14180) 22 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:21.021 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: audit 2024-01-13T09:47:18.643041+0000 mon.smithi084 (mon 2024-01-13T09:47:21.021 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: .0) 178 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2024-01-13T09:47:21.021 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: audit 2024-01-13T09:47:19.643093+0000 mon.smithi084 2024-01-13T09:47:21.021 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: (mon.0) 179 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2024-01-13T09:47:21.021 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: cluster 2024-01-13T09:47:20 2024-01-13T09:47:21.021 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: .065545+0000 mgr.smithi084.itrhok (mgr.14180) 23 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:21.021 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: audit 2024-01-13T09:47 2024-01-13T09:47:21.021 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: :20.642995+0000 mon.smithi084 (mon.0) 180 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2024-01-13T09:47:21.021 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: cluster 2024-01-13T 2024-01-13T09:47:21.022 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: 09:47:20.672549+0000 mon.smithi084 (mon.0) 181 : cluster [INF] mon.smithi084 is new leader, mons smithi084,smithi102 in quorum (ranks 0,1) 2024-01-13T09:47:21.022 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: cluster 2024-01-13T09 2024-01-13T09:47:21.022 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: :47:20.680978+0000 mon.smithi084 (mon.0) 182 : cluster [DBG] monmap e2: 2 mons at {smithi084=[v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0],smithi102=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0]} removed_ranks: {} 2024-01-13T09:47:21.022 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: cluster 2024-01-13T09: 2024-01-13T09:47:21.022 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: 47:20.693055+0000 mon.smithi084 (mon.0) 183 : cluster [DBG] fsmap 2024-01-13T09:47:21.022 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: cluster 2024-01-13 2024-01-13T09:47:21.022 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: T09:47:20.693113+0000 mon.smithi084 (mon.0) 184 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-01-13T09:47:21.022 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: cluster 2024-01-13T09:47:21.022 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: 2024-01-13T09:47:20.693344+0000 mon.smithi084 (mon.0) 185 : cluster [DBG] mgrmap e17: smithi084.itrhok(active, since 38s) 2024-01-13T09:47:21.022 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: cluster 2024-01-13T09:47:20.694349+0000 mon.smithi084 (mon.0) 186 : cluster [DBG] 2024-01-13T09:47:21.022 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: Standby manager daemon smithi102.yjovaz started 2024-01-13T09:47:21.022 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: audit 2024-01-13T09:47:20.695542+0000 2024-01-13T09:47:21.022 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: mon.smithi084 (mon.0) 187 : audit [DBG] from='mgr.? 172.21.15.102:0/1546808686' entity='mgr.smithi102.yjovaz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-01-13T09:47:21.022 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: cluster 2024-01-13T09: 2024-01-13T09:47:21.022 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: 47:20.697197+0000 mon.smithi084 (mon.0) 188 : cluster [INF] overall HEALTH_OK 2024-01-13T09:47:21.022 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: audit 2024-01- 2024-01-13T09:47:21.023 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: 13T09:47:20.697648+0000 mon.smithi084 (mon.0) 189 : audit [DBG] from='mgr.? 172.21.15.102:0/1546808686' entity='mgr.smithi102.yjovaz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi102.yjovaz/key"}]: dispatch 2024-01-13T09:47:21.023 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: audit 2024-01-13T09:47:21.023 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: 2024-01-13T09:47:20.698063+0000 mon.smithi084 (mon.0) 190 : audit [DBG] from='mgr.? 172.21.15.102:0/1546808686' entity='mgr.smithi102.yjovaz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-01-13T09:47:21.023 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: audit 2024-01-13T09:47:21.023 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: 2024-01-13T09:47:20.701524+0000 mon.smithi084 (mon.0) 191 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:21.023 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: cephadm 2024-01 2024-01-13T09:47:21.023 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:20 smithi102 conmon[37544]: -13T09:47:20.703658+0000 mgr.smithi084.itrhok (mgr.14180) 24 : cephadm [INF] Deploying daemon node-exporter.smithi102 on smithi102 2024-01-13T09:47:21.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: audit 2024-01-13T09:47:15.646516+0000 mon.smithi084 (mon.0) 172 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mon metadata", "id": "smithi084"}]: dispatch 2024-01-13T09:47:21.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: cluster 2024-01-13T09:47:15.646657+0000 mon.smithi084 (mon.0) 173 : cluster [INF] mon.smithi084 calling monitor election 2024-01-13T09:47:21.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: audit 2024-01-13T09:47: 2024-01-13T09:47:21.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: 15.659575+0000 mon.smithi084 (mon.0) 174 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2024-01-13T09:47:21.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: cluster 2024-01- 2024-01-13T09:47:21.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: 13T09:47:16.064802+0000 mgr.smithi084.itrhok (mgr.14180) 21 : cluster 2024-01-13T09:47:21.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:21.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: audit 2024 2024-01-13T09:47:21.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: -01-13T09:47:16.642618 2024-01-13T09:47:21.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: +0000 mon.smithi084 (mon.0) 175 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2024-01-13T09:47:21.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: 2024-01-13T09:47:21.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: audit 2024-01-13T09:47:21.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: 2024-01-13T09:47:17. 2024-01-13T09:47:21.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: 152227+0000 mon.smithi084 (mon.0) 176 : audit [DBG] 2024-01-13T09:47:21.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: from='mgr.? 172.21.15.102:0/1546808686' entity='mgr.smithi102.yjovaz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi102.yjovaz/crt"}]: dispatch 2024-01-13T09:47:21.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: audit 2024-01-13T 2024-01-13T09:47:21.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: 09:47:17.642674+0000 mon.smithi084 ( 2024-01-13T09:47:21.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: mon.0) 177 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2024-01-13T09:47:21.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: cluster 2024-01-13T09:47:21.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: 2024-01-13 2024-01-13T09:47:21.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: T09:47:17.644939+0000 mon.smithi102 (mon 2024-01-13T09:47:21.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: .1) 1 : cluster [INF] mon.smithi102 calling monitor election 2024-01-13T09:47:21.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: cluster 2024-01-13T09:47:21.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: 2024-01-13T09:47: 2024-01-13T09:47:21.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: 18.065170+0000 mgr.smithi084.itrhok (mgr.14180) 22 : cluster 2024-01-13T09:47:21.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:21.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: audit 2024 2024-01-13T09:47:21.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: -01-13T09:47:18.643041+0000 2024-01-13T09:47:21.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: mon.smithi084 (mon.0) 178 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2024-01-13T09:47:21.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: audit 2024-01-13T09:47:21.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: 2024-01-13T09 2024-01-13T09:47:21.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: :47:19.643093+0000 mon.smithi084 (mon.0) 179 2024-01-13T09:47:21.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2024-01-13T09:47:21.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: cluster 2024- 2024-01-13T09:47:21.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: 01-13T09:47:20.065545+0000 2024-01-13T09:47:21.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: mgr.smithi084.itrhok (mgr.14180) 23 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:21.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: audit 2024-01-13T09:47:21.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: 2024-01-13T 2024-01-13T09:47:21.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: 09:47:20.642995+0000 mon.smithi084 (mon. 2024-01-13T09:47:21.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: 0) 180 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2024-01-13T09:47:21.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: cluster 2024-01-13T09:47:21.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: 2024-01-13T09:47:20 2024-01-13T09:47:21.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: .672549+0000 mon.smithi084 (mon.0) 181 : cluster 2024-01-13T09:47:21.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: [INF] mon.smithi084 is new leader, mons smithi084,smithi102 in quorum (ranks 0,1) 2024-01-13T09:47:21.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: cluster 2024-01-13T09:47:20. 2024-01-13T09:47:21.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: 680978+0000 mon.smithi084 (mon.0) 182 : cluster 2024-01-13T09:47:21.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: [DBG] monmap e2: 2 mons at {smithi084=[v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0],smithi102=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0]} removed_ranks: {} 2024-01-13T09:47:21.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: cluster 2024-01-13T09:47:21.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: 2024-01-13T09: 2024-01-13T09:47:21.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: 47:20.693055+0000 mon.smithi084 (mon.0) 2024-01-13T09:47:21.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: 183 : cluster [DBG] fsmap 2024-01-13T09:47:21.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: cluster 2024 2024-01-13T09:47:21.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: -01-13T09:47:20 2024-01-13T09:47:21.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: .693113+0000 mon.smithi084 (mon.0) 184 : cluster 2024-01-13T09:47:21.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-01-13T09:47:21.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: cluster 2024-01-13 2024-01-13T09:47:21.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: T09:47:20.693344+0000 mon.smithi084 ( 2024-01-13T09:47:21.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: mon.0) 185 : cluster [DBG] mgrmap e17: smithi084.itrhok(active, since 38s) 2024-01-13T09:47:21.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: cluster 2024-01-13T09:47:21.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: 2024-01-13T 2024-01-13T09:47:21.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: 09:47:20.694349+0000 mon.smithi084 (mon.0 2024-01-13T09:47:21.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: ) 186 : cluster [DBG] Standby manager daemon smithi102.yjovaz started 2024-01-13T09:47:21.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: audit 2024-01-13T09:47:21.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: 2024-01-13T09: 2024-01-13T09:47:21.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: 47:20.695542+0000 mon.smithi084 (mon.0) 187 2024-01-13T09:47:21.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: : audit [DBG] from='mgr.? 172.21.15.102:0/1546808686' entity='mgr.smithi102.yjovaz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-01-13T09:47:21.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: cluster 2024-01- 2024-01-13T09:47:21.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: 13T09:47:20.697197+0000 mon.smithi084 ( 2024-01-13T09:47:21.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: mon.0) 188 : cluster [INF] overall HEALTH_OK 2024-01-13T09:47:21.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: audit 2024-01-13T09:47:21.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: 2024-01-13T 2024-01-13T09:47:21.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: 09:47:20.697648+0000 mon.smithi084 (mon.0 2024-01-13T09:47:21.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: ) 189 : audit [DBG] from='mgr.? 172.21.15.102:0/1546808686' entity='mgr.smithi102.yjovaz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi102.yjovaz/key"}]: dispatch 2024-01-13T09:47:21.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: audit 2024-01-13T09:47:21.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: 2024-01-13T09: 2024-01-13T09:47:21.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: 47:20.698063+0000 mon.smithi084 (mon.0) 190 2024-01-13T09:47:21.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: : audit [DBG] from='mgr.? 172.21.15.102:0/1546808686' entity='mgr.smithi102.yjovaz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-01-13T09:47:21.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: audit 2024-01- 2024-01-13T09:47:21.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: 13T09:47:20.701524+0000 mon.smithi084 (mon 2024-01-13T09:47:21.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: .0) 191 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:21.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: cephadm 2024-01-13T09:47:21.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: 2024-01-13T09:47:20 2024-01-13T09:47:21.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: .703658+0000 mgr.smithi084.itrhok (mgr.14180) 24 : cephadm 2024-01-13T09:47:21.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:20 smithi084 conmon[33408]: [INF] Deploying daemon node-exporter.smithi102 on smithi102 2024-01-13T09:47:21.197 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-01-13T09:47:21.197 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":2,"fsid":"503cf380-b1f8-11ee-95ac-87774f69a715","modified":"2024-01-13T09:47:15.644006Z","created":"2024-01-13T09:44:57.585744Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi102","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-01-13T09:47:21.198 INFO:teuthology.orchestra.run.smithi102.stderr:dumped monmap epoch 2 2024-01-13T09:47:21.711 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-01-13T09:47:21.711 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph config generate-minimal-conf 2024-01-13T09:47:21.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:21 smithi102 conmon[37544]: cluster 2024-01-13T 2024-01-13T09:47:21.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:21 smithi102 conmon[37544]: 09:47:20.757563+0000 mon.smithi084 (mon.0) 192 : cluster [DBG] mgrmap e18: smithi084.itrhok(active, since 38s), standbys: smithi102.yjovaz 2024-01-13T09:47:21.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:21 smithi102 conmon[37544]: audit 2024-01-13T09:47:20.757715+0000 mon.smithi084 (mon.0) 193 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mgr metadata", "who": "smithi102.yjovaz", "id": "smithi102.yjovaz"}]: dispatch 2024-01-13T09:47:21.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:21 smithi102 conmon[37544]: audit 2024-01-13T09:47:21.205814+0000 mon.smithi084 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.102:0/3048042956' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T09:47:21.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:21 smithi102 conmon[37544]: audit 2024-01-13T09:47:21.643344+0000 mon.smithi084 (mon.0) 195 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2024-01-13T09:47:22.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:21 smithi084 conmon[33408]: cluster 2024-01-13T09:47:20.757563+0000 mon.smithi084 (mon.0) 192 : cluster [DBG] mgrmap e18: smithi084.itrhok(active, since 38s), standbys: smithi102.yjovaz 2024-01-13T09:47:22.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:21 smithi084 conmon[33408]: audit 2024-01-13T09:47:20.757715+0000 mon.smithi084 (mon.0) 193 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mgr metadata", "who": "smithi102.yjovaz", "id": "smithi102.yjovaz"}]: dispatch 2024-01-13T09:47:22.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:21 smithi084 conmon[33408]: audit 2024-01-13T09:47:21.205814+0000 mon.smithi084 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.102:0/3048042956' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T09:47:22.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:21 smithi084 conmon[33408]: audit 2024-01-13T09:47:21.643344+0000 mon.smithi084 (mon.0) 195 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mon metadata", "id": "smithi102"}]: dispatch 2024-01-13T09:47:23.373 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:23 smithi084 conmon[33408]: cluster 2024-01-13T09:47:22.065873+0000 mgr.smithi084.itrhok (mgr.14180) 25 : 2024-01-13T09:47:23.374 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:23 smithi084 conmon[33408]: cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:23.374 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:23 smithi084 conmon[33408]: audit 2024-01-13T09:47:22.211580+0000 mon.smithi084 (mon.0) 196 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:23.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:23 smithi102 conmon[37544]: cluster 2024-01-13T09:47:22 2024-01-13T09:47:23.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:23 smithi102 conmon[37544]: .065873+0000 mgr.smithi084.itrhok (mgr.14180) 25 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:23.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:23 smithi102 conmon[37544]: audit 2024-01 2024-01-13T09:47:23.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:23 smithi102 conmon[37544]: -13T09:47:22.211580+0000 mon.smithi084 (mon.0) 196 : audit [INF] 2024-01-13T09:47:23.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:23 smithi102 conmon[37544]: from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:23.741 INFO:teuthology.orchestra.run.smithi084.stdout:# minimal ceph.conf for 503cf380-b1f8-11ee-95ac-87774f69a715 2024-01-13T09:47:23.741 INFO:teuthology.orchestra.run.smithi084.stdout:[global] 2024-01-13T09:47:23.741 INFO:teuthology.orchestra.run.smithi084.stdout: fsid = 503cf380-b1f8-11ee-95ac-87774f69a715 2024-01-13T09:47:23.741 INFO:teuthology.orchestra.run.smithi084.stdout: mon_host = [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] 2024-01-13T09:47:24.273 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-01-13T09:47:24.273 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-01-13T09:47:24.273 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/ceph/ceph.conf 2024-01-13T09:47:24.303 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-01-13T09:47:24.303 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-13T09:47:24.371 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-01-13T09:47:24.371 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/ceph.conf 2024-01-13T09:47:24.402 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-01-13T09:47:24.403 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-13T09:47:24.468 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:24 smithi102 conmon[37544]: audit 2024-01-13T09:47:23.751094+0000 mon.smithi084 (mon.0) 197 : audit 2024-01-13T09:47:24.468 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:24 smithi102 conmon[37544]: [DBG] from='client.? 172.21.15.84:0/2983200151' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:47:24.472 INFO:tasks.cephadm:Deploying OSDs... 2024-01-13T09:47:24.473 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-01-13T09:47:24.473 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/scratch_devs of=/dev/stdout 2024-01-13T09:47:24.491 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-01-13T09:47:24.492 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/nvme1n1 2024-01-13T09:47:24.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:24 smithi084 conmon[33408]: audit 2024-01-13T09:47:23.751094+0000 mon.smithi084 (mon.0) 197 : audit [DBG] from='client.? 172.21.15.84:0/2983200151' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:47:24.544 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/nvme1n1 2024-01-13T09:47:24.544 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-13T09:47:24.544 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 6h/6d Inode: 189358 Links: 1 Device type: 103,1 2024-01-13T09:47:24.544 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-13T09:47:24.544 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:nvme_device_t:s0 2024-01-13T09:47:24.544 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-01-13 09:46:02.321786753 +0000 2024-01-13T09:47:24.544 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-01-13 09:44:12.710694678 +0000 2024-01-13T09:47:24.544 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-01-13 09:44:12.710694678 +0000 2024-01-13T09:47:24.544 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: - 2024-01-13T09:47:24.545 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-01-13T09:47:24.610 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-01-13T09:47:24.610 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-01-13T09:47:24.610 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000244341 s, 2.1 MB/s 2024-01-13T09:47:24.612 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-01-13T09:47:24.670 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/nvme2n1 2024-01-13T09:47:24.727 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/nvme2n1 2024-01-13T09:47:24.727 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-13T09:47:24.727 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 6h/6d Inode: 189425 Links: 1 Device type: 103,2 2024-01-13T09:47:24.727 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-13T09:47:24.727 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:nvme_device_t:s0 2024-01-13T09:47:24.727 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-01-13 09:46:02.332786563 +0000 2024-01-13T09:47:24.728 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-01-13 09:44:12.861692012 +0000 2024-01-13T09:47:24.728 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-01-13 09:44:12.861692012 +0000 2024-01-13T09:47:24.728 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: - 2024-01-13T09:47:24.728 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-01-13T09:47:24.792 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-01-13T09:47:24.793 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-01-13T09:47:24.793 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000252841 s, 2.0 MB/s 2024-01-13T09:47:24.794 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-01-13T09:47:24.852 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/nvme3n1 2024-01-13T09:47:24.911 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/nvme3n1 2024-01-13T09:47:24.911 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-13T09:47:24.911 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 6h/6d Inode: 200762 Links: 1 Device type: 103,3 2024-01-13T09:47:24.912 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-13T09:47:24.912 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:nvme_device_t:s0 2024-01-13T09:47:24.912 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-01-13 09:46:02.343786374 +0000 2024-01-13T09:47:24.912 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-01-13 09:44:13.009689399 +0000 2024-01-13T09:47:24.912 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-01-13 09:44:13.009689399 +0000 2024-01-13T09:47:24.913 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: - 2024-01-13T09:47:24.913 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-01-13T09:47:24.977 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-01-13T09:47:24.977 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-01-13T09:47:24.977 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000211535 s, 2.4 MB/s 2024-01-13T09:47:24.979 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-01-13T09:47:25.037 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/nvme4n1 2024-01-13T09:47:25.095 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/nvme4n1 2024-01-13T09:47:25.095 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-13T09:47:25.095 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 6h/6d Inode: 200814 Links: 1 Device type: 103,4 2024-01-13T09:47:25.095 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-13T09:47:25.095 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:nvme_device_t:s0 2024-01-13T09:47:25.095 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-01-13 09:46:02.354786184 +0000 2024-01-13T09:47:25.095 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-01-13 09:44:13.158686768 +0000 2024-01-13T09:47:25.095 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-01-13 09:44:13.158686768 +0000 2024-01-13T09:47:25.095 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: - 2024-01-13T09:47:25.096 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-01-13T09:47:25.160 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-01-13T09:47:25.160 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-01-13T09:47:25.160 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000256592 s, 2.0 MB/s 2024-01-13T09:47:25.161 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-01-13T09:47:25.220 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-01-13T09:47:25.220 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/scratch_devs of=/dev/stdout 2024-01-13T09:47:25.237 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-01-13T09:47:25.237 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/nvme1n1 2024-01-13T09:47:25.295 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/nvme1n1 2024-01-13T09:47:25.295 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-13T09:47:25.295 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 6h/6d Inode: 198731 Links: 1 Device type: 103,1 2024-01-13T09:47:25.295 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-13T09:47:25.295 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:nvme_device_t:s0 2024-01-13T09:47:25.295 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-01-13 09:47:03.729956858 +0000 2024-01-13T09:47:25.295 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-01-13 09:44:14.419311996 +0000 2024-01-13T09:47:25.295 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-01-13 09:44:14.419311996 +0000 2024-01-13T09:47:25.295 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: - 2024-01-13T09:47:25.295 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-01-13T09:47:25.362 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-01-13T09:47:25.362 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-01-13T09:47:25.363 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000297386 s, 1.7 MB/s 2024-01-13T09:47:25.363 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-01-13T09:47:25.422 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/nvme2n1 2024-01-13T09:47:25.479 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/nvme2n1 2024-01-13T09:47:25.479 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-13T09:47:25.480 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 6h/6d Inode: 199802 Links: 1 Device type: 103,2 2024-01-13T09:47:25.480 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-13T09:47:25.480 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:nvme_device_t:s0 2024-01-13T09:47:25.480 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-01-13 09:47:03.740956640 +0000 2024-01-13T09:47:25.480 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-01-13 09:44:14.574308917 +0000 2024-01-13T09:47:25.480 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-01-13 09:44:14.574308917 +0000 2024-01-13T09:47:25.480 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: - 2024-01-13T09:47:25.480 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-01-13T09:47:25.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:25 smithi084 conmon[33408]: cluster 2024-01-13T09:47:24.066163+0000 mgr.smithi084.itrhok (mgr.14180) 2024-01-13T09:47:25.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:25 smithi084 conmon[33408]: 26 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:25.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:25 smithi084 conmon[33408]: audit 2024-01-13T09:47:24.267480+0000 mon.smithi084 (mon.0) 198 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:25.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:25 smithi084 conmon[33408]: audit 2024-01-13T09:47:24.269046+0000 mon.smithi084 (mon.0) 199 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T09:47:25.543 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:25 smithi102 conmon[37544]: cluster 2024-01-13T09:47:24.066163+0000 mgr.smithi084.itrhok (mgr.14180) 26 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:25.543 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:25 smithi102 conmon[37544]: audit 2024-01-13T 2024-01-13T09:47:25.543 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:25 smithi102 conmon[37544]: 09:47:24.267480+0000 mon.smithi084 (mon.0) 198 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:25.543 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:25 smithi102 conmon[37544]: audit 2024-01-13T09:47:24.269046+0000 mon.smithi084 (mon.0) 199 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T09:47:25.545 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-01-13T09:47:25.545 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-01-13T09:47:25.545 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000271456 s, 1.9 MB/s 2024-01-13T09:47:25.546 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-01-13T09:47:25.605 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/nvme3n1 2024-01-13T09:47:25.662 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/nvme3n1 2024-01-13T09:47:25.662 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-13T09:47:25.663 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 6h/6d Inode: 195018 Links: 1 Device type: 103,3 2024-01-13T09:47:25.663 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-13T09:47:25.663 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:nvme_device_t:s0 2024-01-13T09:47:25.663 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-01-13 09:47:03.752956403 +0000 2024-01-13T09:47:25.663 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-01-13 09:44:14.728305857 +0000 2024-01-13T09:47:25.663 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-01-13 09:44:14.728305857 +0000 2024-01-13T09:47:25.663 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: - 2024-01-13T09:47:25.663 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-01-13T09:47:25.728 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-01-13T09:47:25.729 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-01-13T09:47:25.729 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000188905 s, 2.7 MB/s 2024-01-13T09:47:25.730 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-01-13T09:47:25.788 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/nvme4n1 2024-01-13T09:47:25.845 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/nvme4n1 2024-01-13T09:47:25.845 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-13T09:47:25.845 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 6h/6d Inode: 195190 Links: 1 Device type: 103,4 2024-01-13T09:47:25.845 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-13T09:47:25.845 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:nvme_device_t:s0 2024-01-13T09:47:25.845 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-01-13 09:47:03.763956185 +0000 2024-01-13T09:47:25.846 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-01-13 09:44:14.882302797 +0000 2024-01-13T09:47:25.846 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-01-13 09:44:14.882302797 +0000 2024-01-13T09:47:25.846 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: - 2024-01-13T09:47:25.846 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-01-13T09:47:25.910 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-01-13T09:47:25.911 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-01-13T09:47:25.911 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.00024686 s, 2.1 MB/s 2024-01-13T09:47:25.912 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-01-13T09:47:25.970 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph orch apply osd --all-available-devices 2024-01-13T09:47:26.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:26 smithi102 conmon[37544]: audit 2024-01-13T09:47:25.608993+0000 mon.smithi084 (mon.0) 200 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:26.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:26 smithi102 conmon[37544]: audit 2024-01-13T09:47:25.609717+0000 mon.smithi084 (mon.0) 201 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:47:26.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:26 smithi102 conmon[37544]: audit 2024-01-13T09:47:25.610371+0000 mon.smithi084 (mon.0) 202 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T09:47:26.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:26 smithi102 conmon[37544]: cephadm 2024-01-13T09:47:25.611342+0000 mgr.smithi084.itrhok (mgr.14180) 27 : cephadm [INF] Updating smithi084:/etc/ceph/ceph.conf 2024-01-13T09:47:26.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:26 smithi102 conmon[37544]: cephadm 2024-01-13T09:47:25.611610+0000 2024-01-13T09:47:26.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:26 smithi102 conmon[37544]: mgr.smithi084.itrhok (mgr.14180) 28 : cephadm [INF] Updating smithi102:/etc/ceph/ceph.conf 2024-01-13T09:47:26.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:26 smithi102 conmon[37544]: audit 2024-01-13T09:47:25.630195+0000 mon.smithi084 (mon.0) 203 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:26.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:26 smithi102 conmon[37544]: audit 2024-01-13T09:47:25.651071+0000 mon.smithi084 (mon.0) 204 : audit [INF] 2024-01-13T09:47:26.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:26 smithi102 conmon[37544]: from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:26.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:26 smithi102 conmon[37544]: audit 2024-01 2024-01-13T09:47:26.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:26 smithi102 conmon[37544]: -13T09:47:25.654433+0000 mon.smithi084 2024-01-13T09:47:26.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:26 smithi102 conmon[37544]: (mon.0) 205 : audit 2024-01-13T09:47:26.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:26 smithi102 conmon[37544]: [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:26.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:26 smithi102 conmon[37544]: cephadm 2024-01-13T09:47:26.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:26 smithi102 conmon[37544]: 2024-01-13T09 2024-01-13T09:47:26.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:26 smithi102 conmon[37544]: :47:25.657816+0000 2024-01-13T09:47:26.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:26 smithi102 conmon[37544]: mgr.smithi084.itrhok (mgr.14180) 29 : cephadm 2024-01-13T09:47:26.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:26 smithi102 conmon[37544]: [INF] Reconfiguring mon.smithi084 (unknown last config time)... 2024-01-13T09:47:26.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:26 smithi102 conmon[37544]: audit 2024-01-13T09:47:26.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:26 smithi102 conmon[37544]: 2024-01-13T09 2024-01-13T09:47:26.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:26 smithi102 conmon[37544]: :47:25.658116+0000 mon.smithi084 (mon. 2024-01-13T09:47:26.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:26 smithi102 conmon[37544]: 0) 206 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-13T09:47:26.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:26 smithi102 conmon[37544]: audit 2024-01-13T09:47:26.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:26 smithi102 conmon[37544]: 2024-01 2024-01-13T09:47:26.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:26 smithi102 conmon[37544]: -13T09:47:25. 2024-01-13T09:47:26.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:26 smithi102 conmon[37544]: 658688+0000 mon.smithi084 (mon.0 2024-01-13T09:47:26.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:26 smithi102 conmon[37544]: ) 207 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-13T09:47:26.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:26 smithi102 conmon[37544]: audit 2024-01 2024-01-13T09:47:26.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:26 smithi102 conmon[37544]: -13T09:47:25. 2024-01-13T09:47:26.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:26 smithi102 conmon[37544]: 659203+0000 mon.smithi084 (mon.0) 2024-01-13T09:47:26.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:26 smithi102 conmon[37544]: 208 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:47:26.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:26 smithi102 conmon[37544]: cephadm 2024-01-13T09:47:26.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:26 smithi102 conmon[37544]: 2024-01-13T09 2024-01-13T09:47:26.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:26 smithi102 conmon[37544]: :47:25.659698+0000 mgr.smithi084.itrhok 2024-01-13T09:47:26.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:26 smithi102 conmon[37544]: (mgr.14180) 30 : cephadm 2024-01-13T09:47:26.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:26 smithi102 conmon[37544]: [INF] Reconfiguring daemon mon.smithi084 on smithi084 2024-01-13T09:47:26.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:26 smithi084 conmon[33408]: audit 2024-01-13T09:47:25.608993+0000 mon.smithi084 (mon.0) 200 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:26.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:26 smithi084 conmon[33408]: audit 2024-01-13T09:47:25.609717+0000 mon.smithi084 (mon.0) 201 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:47:26.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:26 smithi084 conmon[33408]: audit 2024-01-13T09:47:25.610371+0000 mon.smithi084 (mon.0) 202 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T09:47:26.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:26 smithi084 conmon[33408]: cephadm 2024-01-13T09:47:25.611342+0000 mgr.smithi084.itrhok (mgr.14180) 27 : cephadm [INF] Updating smithi084:/etc/ceph/ceph.conf 2024-01-13T09:47:26.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:26 smithi084 conmon[33408]: cephadm 2024-01-13T09:47:25.611610+0000 mgr.smithi084.itrhok (mgr.14180) 28 : cephadm [INF] Updating smithi102:/etc/ceph/ceph.conf 2024-01-13T09:47:26.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:26 smithi084 conmon[33408]: audit 2024-01-13T09:47:25.630195+0000 mon.smithi084 (mon.0) 203 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:26.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:26 smithi084 conmon[33408]: audit 2024-01-13T09:47:25.651071+0000 mon.smithi084 (mon.0) 204 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:26.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:26 smithi084 conmon[33408]: audit 2024-01-13T09:47:25.654433+0000 mon.smithi084 (mon.0) 205 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:26.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:26 smithi084 conmon[33408]: cephadm 2024-01-13T09:47:25.657816+0000 mgr.smithi084.itrhok (mgr.14180) 29 : cephadm [INF] Reconfiguring mon.smithi084 (unknown last config time)... 2024-01-13T09:47:26.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:26 smithi084 conmon[33408]: audit 2024-01-13T09:47:25.658116+0000 mon.smithi084 (mon.0) 206 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-13T09:47:26.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:26 smithi084 conmon[33408]: audit 2024-01-13T09:47:25.658688+0000 mon.smithi084 (mon.0) 207 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-13T09:47:26.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:26 smithi084 conmon[33408]: audit 2024-01-13T09:47:25.659203+0000 mon.smithi084 (mon.0) 208 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:47:26.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:26 smithi084 conmon[33408]: cephadm 2024-01-13T09:47:25.659698+0000 mgr.smithi084.itrhok (mgr.14180) 30 : cephadm [INF] Reconfiguring daemon mon.smithi084 on smithi084 2024-01-13T09:47:27.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:27 smithi084 conmon[33408]: cluster 2024-01-13T09:47:26.066443+0000 mgr.smithi084.itrhok (mgr.14180) 31 : cluster 2024-01-13T09:47:27.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:27 smithi084 conmon[33408]: [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:27.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:27 smithi084 conmon[33408]: audit 2024-01-13T09:47:26.997554+0000 mon.smithi084 (mon.0) 209 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:27.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:27 smithi084 conmon[33408]: audit 2024-01-13T09:47:26.998348+0000 mon.smithi084 (mon.0) 210 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi084.itrhok", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-13T09:47:27.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:27 smithi084 conmon[33408]: audit 2024-01-13T09:47:26.999053+0000 mon.smithi084 (mon.0) 211 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-13T09:47:27.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:27 smithi084 conmon[33408]: audit 2024-01-13T09:47:26.999617+0000 mon.smithi084 (mon.0) 212 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:47:27.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:27 smithi084 conmon[33408]: audit 2024-01-13T09:47:27.215566+0000 mon.smithi084 (mon.0) 213 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:27.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:27 smithi102 conmon[37544]: cluster 2024-01-13T09:47:26.066443+0000 mgr.smithi084.itrhok (mgr.14180) 2024-01-13T09:47:27.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:27 smithi102 conmon[37544]: 31 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:27.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:27 smithi102 conmon[37544]: audit 2024-01-13T09:47:26.997554+0000 mon.smithi084 (mon.0) 209 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:27.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:27 smithi102 conmon[37544]: audit 2024-01-13T09:47:26.998348+0000 mon.smithi084 (mon.0) 210 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi084.itrhok", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-13T09:47:27.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:27 smithi102 conmon[37544]: audit 2024-01-13T09:47:26.999053+0000 mon.smithi084 (mon.0) 211 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-13T09:47:27.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:27 smithi102 conmon[37544]: audit 2024-01-13T09:47:26.999617+0000 mon.smithi084 (mon.0) 212 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:47:27.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:27 smithi102 conmon[37544]: audit 2024-01-13T09:47:27.215566+0000 mon.smithi084 (mon.0) 213 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:27.988 INFO:teuthology.orchestra.run.smithi102.stdout:Scheduled osd.all-available-devices update... 2024-01-13T09:47:28.535 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-01-13T09:47:28.536 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:47:28.874 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:28 smithi084 conmon[33408]: cephadm 2024-01-13T09:47:26.998102+0000 mgr.smithi084.itrhok (mgr.14180) 32 : cephadm [INF] Reconfiguring mgr.smithi084.itrhok (unknown last config time)... 2024-01-13T09:47:28.874 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:28 smithi084 conmon[33408]: cephadm 2024-01-13T09:47:27.000109+0000 mgr.smithi084.itrhok (mgr.14180) 33 : cephadm [INF] Reconfiguring daemon mgr.smithi084.itrhok on smithi084 2024-01-13T09:47:28.874 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:28 smithi084 conmon[33408]: audit 2024-01-13T09:47:27.997420+0000 mon.smithi084 (mon.0) 214 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:28.874 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:28 smithi084 conmon[33408]: audit 2024-01-13T09:47:28.301030+0000 mon.smithi084 (mon.0) 215 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:28.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:28 smithi102 conmon[37544]: cephadm 2024-01-13T09:47:26.998102+0000 mgr.smithi084.itrhok (mgr.14180) 32 : cephadm [INF] Reconfiguring mgr.smithi084.itrhok (unknown last config time)... 2024-01-13T09:47:28.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:28 smithi102 conmon[37544]: cephadm 2024-01-13T09:47:27.000109+0000 mgr.smithi084.itrhok (mgr.14180) 33 : cephadm [INF] Reconfiguring daemon mgr.smithi084.itrhok on smithi084 2024-01-13T09:47:28.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:28 smithi102 conmon[37544]: audit 2024-01-13T09:47:27.997420+0000 mon.smithi084 (mon.0) 214 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:28.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:28 smithi102 conmon[37544]: audit 2024-01-13T09:47:28.301030+0000 mon.smithi084 (mon.0) 215 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:29.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:29 smithi102 conmon[37544]: audit 2024-01-13T09:47:27.992494+0000 mgr.smithi084.itrhok (mgr.14180) 34 2024-01-13T09:47:29.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:29 smithi102 conmon[37544]: : audit [DBG] from='client.24101 -' entity='client.admin' cmd=[{"prefix": "orch apply osd", "all_available_devices": true, "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T09:47:29.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:29 smithi102 conmon[37544]: cephadm 2024-01-13T09:47:27.993817+0000 mgr.smithi084.itrhok (mgr.14180) 35 : cephadm [INF] Marking host: smithi084 for OSDSpec preview refresh. 2024-01-13T09:47:29.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:29 smithi102 conmon[37544]: cephadm 2024-01-13T09:47:27.993886+0000 mgr.smithi084.itrhok (mgr.14180) 36 : cephadm [INF] Marking host: smithi102 for OSDSpec preview refresh. 2024-01-13T09:47:29.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:29 smithi102 conmon[37544]: cephadm 2024-01-13T09:47:27.994142+0000 mgr.smithi084.itrhok (mgr.14180) 37 : cephadm [INF] Saving service osd.all-available-devices spec with placement * 2024-01-13T09:47:29.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:29 smithi102 conmon[37544]: cluster 2024-01-13T09:47:28.066686+0000 mgr.smithi084.itrhok (mgr.14180) 38 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:29.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:29 smithi102 conmon[37544]: cephadm 2024-01-13T09: 2024-01-13T09:47:29.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:29 smithi102 conmon[37544]: 47:28.301677+0000 mgr.smithi084.itrhok (mgr.14180) 39 : cephadm [INF] 2024-01-13T09:47:29.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:29 smithi102 conmon[37544]: Reconfiguring alertmanager.smithi084 (dependencies changed)... 2024-01-13T09:47:29.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:29 smithi102 conmon[37544]: cephadm 2024-01-13T09:47:28.307481 2024-01-13T09:47:29.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:29 smithi102 conmon[37544]: +0000 mgr.smithi084.itrhok (mgr.14180) 40 : cephadm [INF] Reconfiguring daemon alertmanager.smithi084 on smithi084 2024-01-13T09:47:29.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:29 smithi102 conmon[37544]: 2024-01-13T09:47:30.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:29 smithi084 conmon[33408]: audit 2024-01-13T09:47:27.992494+0000 mgr.smithi084.itrhok (mgr.14180) 34 : 2024-01-13T09:47:30.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:29 smithi084 conmon[33408]: audit [DBG] from='client.24101 -' entity='client.admin' cmd=[{"prefix": "orch apply osd", "all_available_devices": true, "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T09:47:30.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:29 smithi084 conmon[33408]: cephadm 2024-01-13T09:47:27.993817+0000 mgr.smithi084.itrhok (mgr.14180) 35 : cephadm [INF] Marking host: smithi084 for OSDSpec preview refresh. 2024-01-13T09:47:30.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:29 smithi084 conmon[33408]: cephadm 2024-01-13T09:47:27.993886+0000 mgr.smithi084.itrhok (mgr.14180) 36 : cephadm [INF] Marking host: smithi102 for OSDSpec preview refresh. 2024-01-13T09:47:30.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:29 smithi084 conmon[33408]: cephadm 2024-01-13T09:47:27.994142+0000 mgr.smithi084.itrhok (mgr.14180) 37 : cephadm [INF] Saving service osd.all-available-devices spec with placement * 2024-01-13T09:47:30.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:29 smithi084 conmon[33408]: cluster 2024-01-13T09:47:28.066686+0000 mgr.smithi084.itrhok (mgr.14180) 38 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:30.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:29 smithi084 conmon[33408]: cephadm 2024-01-13T09:47:28.301677+0000 mgr.smithi084.itrhok (mgr.14180) 39 : cephadm [INF] Reconfiguring alertmanager.smithi084 (dependencies changed)... 2024-01-13T09:47:30.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:29 smithi084 conmon[33408]: cephadm 2024-01-13T09:47:28.307481+0000 mgr.smithi084.itrhok (mgr.14180) 40 : cephadm [INF] Reconfiguring daemon alertmanager.smithi084 on smithi084 2024-01-13T09:47:31.653 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:47:31.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:31 smithi102 conmon[37544]: cluster 2024-01-13T09:47:30 2024-01-13T09:47:31.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:31 smithi102 conmon[37544]: .066970+0000 mgr.smithi084.itrhok (mgr.14180) 41 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:32.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:31 smithi084 conmon[33408]: cluster 2024-01-13T09:47:30.066970+0000 mgr.smithi084.itrhok (mgr.14180) 41 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:32.609 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2024-01-13T09:47:32.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:32 smithi102 conmon[37544]: audit 2024-01-13T09:47:31.663181+0000 2024-01-13T09:47:32.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:32 smithi102 conmon[37544]: mon.smithi084 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.84:0/2900410552' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:47:33.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:32 smithi084 conmon[33408]: audit 2024-01-13T09:47:31.663181+0000 mon.smithi084 (mon.0) 2024-01-13T09:47:33.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:32 smithi084 conmon[33408]: 216 : audit [DBG] from='client.? 172.21.15.84:0/2900410552' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:47:33.610 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:47:33.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:33 smithi102 conmon[37544]: cluster 2024-01-13T09:47:32.067263+0000 mgr.smithi084.itrhok 2024-01-13T09:47:33.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:33 smithi102 conmon[37544]: (mgr.14180) 42 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:34.016 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:33 smithi084 conmon[33408]: cluster 2024-01-13T09:47:32.067263+0000 mgr.smithi084.itrhok (mgr.14180) 42 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:35.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:35 smithi084 conmon[33408]: cluster 2024-01-13T09:47:34.067488+0000 mgr.smithi084.itrhok (mgr 2024-01-13T09:47:35.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:35 smithi084 conmon[33408]: .14180) 43 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:35.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:35 smithi084 conmon[33408]: audit 2024-01-13T09:47:34.259305+0000 mon.smithi084 (mon.0) 217 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:35.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:35 smithi084 conmon[33408]: cephadm 2024-01-13T09:47:34.259874+0000 mgr.smithi084.itrhok (mgr.14180) 44 : cephadm [INF] Reconfiguring crash.smithi084 (monmap changed)... 2024-01-13T09:47:35.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:35 smithi084 conmon[33408]: audit 2024-01-13T09:47:34.260121+0000 mon.smithi084 (mon.0) 218 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi084", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-13T09:47:35.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:35 smithi084 conmon[33408]: audit 2024-01-13T09:47:34.260724+0000 mon.smithi084 (mon.0) 219 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:47:35.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:35 smithi084 conmon[33408]: cephadm 2024-01-13T09:47:34.261168+0000 mgr.smithi084.itrhok (mgr.14180) 45 : cephadm [INF] Reconfiguring daemon crash.smithi084 on smithi084 2024-01-13T09:47:35.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:35 smithi102 conmon[37544]: cluster 2024-01-13T09:47:34.067488+0000 mgr.smithi084.itrhok (mgr. 2024-01-13T09:47:35.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:35 smithi102 conmon[37544]: 14180) 43 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:35.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:35 smithi102 conmon[37544]: audit 2024-01-13T09:47:34.259305+0000 mon.smithi084 (mon.0 2024-01-13T09:47:35.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:35 smithi102 conmon[37544]: ) 217 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:35.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:35 smithi102 conmon[37544]: cephadm 2024-01-13T09:47:35.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:35 smithi102 conmon[37544]: 2024-01-13T09:47:34.259874+0000 mgr.smithi084.itrhok (mgr.14180) 44 : cephadm [INF] Reconfiguring crash.smithi084 (monmap changed)... 2024-01-13T09:47:35.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:35 smithi102 conmon[37544]: 2024-01-13T09:47:35.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:35 smithi102 conmon[37544]: audit 2024-01-13T09:47:34.260121+0000 2024-01-13T09:47:35.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:35 smithi102 conmon[37544]: mon.smithi084 (mon.0) 218 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi084", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-13T09:47:35.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:35 smithi102 conmon[37544]: audit 2024 2024-01-13T09:47:35.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:35 smithi102 conmon[37544]: -01-13T09:47:34.260724+0000 mon.smithi084 (mon. 2024-01-13T09:47:35.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:35 smithi102 conmon[37544]: 0) 219 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:47:35.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:35 smithi102 conmon[37544]: cephadm 2024-01-13T09:47:34.261168+0000 mgr.smithi084.itrhok ( 2024-01-13T09:47:35.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:35 smithi102 conmon[37544]: mgr.14180) 45 : cephadm [INF] Reconfiguring daemon crash.smithi084 on smithi084 2024-01-13T09:47:36.217 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:47:36.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:36 smithi084 conmon[33408]: audit 2024-01-13T09:47:36.227012+0000 mon.smithi084 (mon.0) 220 : 2024-01-13T09:47:36.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:36 smithi084 conmon[33408]: audit [DBG] from='client.? 172.21.15.84:0/2770570352' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:47:36.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:36 smithi102 conmon[37544]: audit 2024-01-13T09:47:36.227012+0000 mon.smithi084 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.84:0/2770570352' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:47:36.844 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2024-01-13T09:47:37.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:37 smithi084 conmon[33408]: cluster 2024-01-13T09:47:36.067856+0000 mgr.smithi084.itrhok (mgr.14180) 46 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:37.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:37 smithi084 conmon[33408]: audit 2024-01-13T09:47:36 2024-01-13T09:47:37.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:37 smithi084 conmon[33408]: .648380+0000 mon.smithi084 (mon.0) 221 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:37.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:37 smithi084 conmon[33408]: cephadm 2024-01-13T09:47:36.649012+0000 mgr.smithi084.itrhok (mgr.14180) 47 : cephadm [INF] Reconfiguring grafana.smithi084 (dependencies changed)... 2024-01-13T09:47:37.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:37 smithi084 conmon[33408]: cephadm 2024-01-13T09:47:36.678908+0000 mgr.smithi084.itrhok (mgr.14180) 48 : cephadm [INF] Reconfiguring daemon grafana.smithi084 on smithi084 2024-01-13T09:47:37.845 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:47:37.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:37 smithi102 conmon[37544]: cluster 2024-01-13T09:47:36.067856+0000 mgr.smithi084.itrhok (mgr.14180) 46 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:37.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:37 smithi102 conmon[37544]: audit 2024-01-13T09:47:36.648380+0000 mon.smithi084 (mon.0) 221 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:37.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:37 smithi102 conmon[37544]: cephadm 2024-01-13T09 2024-01-13T09:47:37.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:37 smithi102 conmon[37544]: :47:36.649012+0000 mgr.smithi084.itrhok (mgr.14180) 47 : cephadm [INF] Reconfiguring grafana.smithi084 (dependencies changed)... 2024-01-13T09:47:37.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:37 smithi102 conmon[37544]: 2024-01-13T09:47:37.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:37 smithi102 conmon[37544]: cephadm 2024-01-13T09:47:37.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:37 smithi102 conmon[37544]: 2024-01-13T09:47:36 2024-01-13T09:47:37.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:37 smithi102 conmon[37544]: .678908+0000 mgr.smithi084.itrhok (mgr.14180) 2024-01-13T09:47:37.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:37 smithi102 conmon[37544]: 48 : cephadm [INF] Reconfiguring daemon grafana.smithi084 on smithi084 2024-01-13T09:47:39.853 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:39 smithi084 conmon[33408]: cluster 2024-01-13T09:47:38.068067+0000 mgr.smithi084.itrhok (mgr.14180) 49 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:39.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:39 smithi102 conmon[37544]: cluster 2024-01-13T09:47:38.068067+0000 mgr.smithi084.itrhok (mgr.14180) 49 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:40.252 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:47:41.439 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2024-01-13T09:47:41.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:41 smithi102 conmon[37544]: cluster 2024-01-13T09:47:40.068326+0000 mgr.smithi084.itrhok (mgr 2024-01-13T09:47:41.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:41 smithi102 conmon[37544]: .14180) 50 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:41.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:41 smithi102 conmon[37544]: audit 2024-01-13T09:47:40.261418+0000 mon.smithi084 (mon 2024-01-13T09:47:41.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:41 smithi102 conmon[37544]: .0) 222 : audit [DBG] from='client.? 172.21.15.84:0/319076238' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:47:41.504 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:41 smithi084 conmon[33408]: cluster 2024-01-13T09:47:40.068326+0000 mgr.smithi084.itrhok (mgr.14180) 2024-01-13T09:47:41.504 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:41 smithi084 conmon[33408]: 50 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:41.504 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:41 smithi084 conmon[33408]: audit 2024-01-13T09:47:40.261418+0000 mon.smithi084 (mon.0) 222 : audit [DBG] from='client.? 172.21.15.84:0/319076238' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:47:42.440 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:47:43.344 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:43 smithi084 conmon[33408]: cluster 2024-01-13T09:47:42.068873+0000 mgr.smithi084.itrhok (mgr.14180) 51 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:43.344 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:43 smithi084 conmon[33408]: audit 2024-01-13T09:47:42.225417+0000 mon.smithi084 (mon.0) 223 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:43.344 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:43 smithi084 conmon[33408]: cephadm 2024-01-13T09:47:42.226206+0000 mgr.smithi084.itrhok (mgr.14180) 52 : cephadm [INF] Reconfiguring prometheus.smithi084 (dependencies changed)... 2024-01-13T09:47:43.344 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:43 smithi084 conmon[33408]: cephadm 2024-01-13T09:47:42.229837+0000 mgr.smithi084.itrhok (mgr.14180) 53 : cephadm [INF] Reconfiguring daemon prometheus.smithi084 on smithi084 2024-01-13T09:47:43.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:43 smithi102 conmon[37544]: cluster 2024-01-13T09:47:42.068873+0000 mgr.smithi084.itrhok (mgr. 2024-01-13T09:47:43.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:43 smithi102 conmon[37544]: 14180) 51 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:43.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:43 smithi102 conmon[37544]: audit 2024-01-13T09:47:42.225417+0000 mon.smithi084 (mon.0) 223 : audit [INF] 2024-01-13T09:47:43.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:43 smithi102 conmon[37544]: from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:43.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:43 smithi102 conmon[37544]: cephadm 2024-01-13T 2024-01-13T09:47:43.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:43 smithi102 conmon[37544]: 09:47:42.226206+0000 mgr.smithi084.itrhok (mgr.14180) 52 : cephadm [INF] Reconfiguring prometheus.smithi084 (dependencies changed)... 2024-01-13T09:47:43.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:43 smithi102 conmon[37544]: 2024-01-13T09:47:43.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:43 smithi102 conmon[37544]: cephadm 2024-01-13T09:47:43.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:43 smithi102 conmon[37544]: 2024-01-13T09:47:42.229837+0000 mgr.smithi084.itrhok (mgr.14180) 53 : 2024-01-13T09:47:43.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:43 smithi102 conmon[37544]: cephadm [INF] Reconfiguring daemon prometheus.smithi084 on smithi084 2024-01-13T09:47:45.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:45 smithi084 conmon[33408]: cluster 2024-01-13T09:47:44.069273+0000 mgr.smithi084.itrhok (mgr.14180) 54 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:45.283 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:47:45.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:45 smithi102 conmon[37544]: cluster 2024-01-13T09:47:44.069273+0000 mgr.smithi084.itrhok (mgr.14180 2024-01-13T09:47:45.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:45 smithi102 conmon[37544]: ) 54 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:46.353 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2024-01-13T09:47:46.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:46 smithi102 conmon[37544]: audit 2024-01-13T09:47:45.293650+0000 mon.smithi084 (mon. 2024-01-13T09:47:46.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:46 smithi102 conmon[37544]: 0) 224 : audit [DBG] from='client.? 172.21.15.84:0/3211796231' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:47:46.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:46 smithi084 conmon[33408]: audit 2024-01-13T09:47:45.293650+0000 mon.smithi084 (mon.0) 224 : 2024-01-13T09:47:46.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:46 smithi084 conmon[33408]: audit [DBG] from='client.? 172.21.15.84:0/3211796231' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:47:47.354 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:47:47.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:47 smithi102 conmon[37544]: cluster 2024-01-13T09:47:46.069608+0000 mgr.smithi084.itrhok (mgr. 2024-01-13T09:47:47.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:47 smithi102 conmon[37544]: 14180) 55 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:47.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:47 smithi084 conmon[33408]: cluster 2024-01-13T09:47:46.069608+0000 mgr.smithi084.itrhok (mgr.14180 2024-01-13T09:47:47.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:47 smithi084 conmon[33408]: ) 55 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:49.161 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:48 smithi102 conmon[37544]: audit 2024-01-13T09:47:47.979004+0000 mon.smithi084 (mon.0) 225 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:49.161 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:48 smithi102 conmon[37544]: cephadm 2024-01-13T09:47:47.979522+0000 mgr.smithi084.itrhok (mgr.14180) 56 : cephadm [INF] Reconfiguring crash.smithi102 (monmap changed)... 2024-01-13T09:47:49.161 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:48 smithi102 conmon[37544]: audit 2024-01-13T09:47:47.979759+0000 mon.smithi084 (mon.0) 226 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi102", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-13T09:47:49.161 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:48 smithi102 conmon[37544]: audit 2024-01-13T09:47:47.980465+0000 mon.smithi084 (mon.0) 227 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:47:49.162 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:48 smithi102 conmon[37544]: cephadm 2024-01-13T09:47:47.980993+0000 mgr.smithi084.itrhok (mgr.14180) 57 : cephadm [INF] Reconfiguring daemon crash.smithi102 on smithi102 2024-01-13T09:47:49.162 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:48 smithi102 conmon[37544]: cluster 2024-01-13T09:47:48.069819+0000 mgr.smithi084.itrhok (mgr.14180) 58 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:49.184 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:48 smithi084 conmon[33408]: audit 2024-01-13T09:47:47.979004+0000 mon.smithi084 (mon.0) 225 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:49.184 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:48 smithi084 conmon[33408]: cephadm 2024-01-13T09:47:47.979522+0000 mgr.smithi084.itrhok (mgr.14180) 56 : cephadm [INF] Reconfiguring crash.smithi102 (monmap changed)... 2024-01-13T09:47:49.184 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:48 smithi084 conmon[33408]: audit 2024-01-13T09:47:47.979759+0000 mon.smithi084 (mon.0) 226 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi102", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-13T09:47:49.184 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:48 smithi084 conmon[33408]: audit 2024-01-13T09:47:47.980465+0000 mon.smithi084 (mon.0) 227 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:47:49.184 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:48 smithi084 conmon[33408]: cephadm 2024-01-13T09:47:47.980993+0000 mgr.smithi084.itrhok (mgr.14180) 57 : cephadm [INF] Reconfiguring daemon crash.smithi102 on smithi102 2024-01-13T09:47:49.184 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:48 smithi084 conmon[33408]: cluster 2024-01-13T09:47:48.069819+0000 mgr.smithi084.itrhok (mgr.14180) 58 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:49.574 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:47:50.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:50 smithi102 conmon[37544]: audit 2024-01-13T09:47:49.205715+0000 mon.smithi084 (mon.0) 228 : audit [INF] 2024-01-13T09:47:50.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:50 smithi102 conmon[37544]: from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:50.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:50 smithi102 conmon[37544]: cephadm 2024-01-13T09:47:49.206228+0000 mgr.smithi084.itrhok (mgr.14180) 59 : cephadm [INF] Reconfiguring mgr.smithi102.yjovaz (monmap changed)... 2024-01-13T09:47:50.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:50 smithi102 conmon[37544]: audit 2024-01-13T09:47:49.206508+0000 mon.smithi084 (mon.0) 229 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi102.yjovaz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-13T09:47:50.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:50 smithi102 conmon[37544]: audit 2024-01-13T09:47:49.207127+0000 mon.smithi084 (mon.0) 230 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-13T09:47:50.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:50 smithi102 conmon[37544]: audit 2024-01-13T09:47:49.207665+0000 mon.smithi084 (mon.0) 231 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:47:50.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:50 smithi102 conmon[37544]: cephadm 2024-01-13T09:47:49.208142+0000 mgr.smithi084.itrhok (mgr.14180) 60 : cephadm 2024-01-13T09:47:50.235 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:50 smithi102 conmon[37544]: [INF] Reconfiguring daemon mgr.smithi102.yjovaz on smithi102 2024-01-13T09:47:50.235 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:50 smithi102 conmon[37544]: audit 2024-01-13T09:47:49.584685+0000 mon.smithi084 (mon.0) 232 : audit [DBG] from='client.? 172.21.15.84:0/2935937377' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:47:50.248 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2024-01-13T09:47:50.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:50 smithi084 conmon[33408]: audit 2024-01-13T09:47:49.205715+0000 mon.smithi084 (mon.0) 228 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:50.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:50 smithi084 conmon[33408]: cephadm 2024-01-13T09:47:49.206228+0000 mgr.smithi084.itrhok (mgr.14180) 59 : cephadm 2024-01-13T09:47:50.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:50 smithi084 conmon[33408]: [INF] Reconfiguring mgr.smithi102.yjovaz (monmap changed)... 2024-01-13T09:47:50.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:50 smithi084 conmon[33408]: audit 2024-01-13T09:47:49.206508+0000 mon.smithi084 (mon.0) 229 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi102.yjovaz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-13T09:47:50.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:50 smithi084 conmon[33408]: audit 2024-01-13T09:47:49.207127+0000 mon.smithi084 (mon.0) 230 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-13T09:47:50.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:50 smithi084 conmon[33408]: audit 2024-01-13T09:47:49.207665+0000 mon.smithi084 (mon.0) 231 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:47:50.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:50 smithi084 conmon[33408]: cephadm 2024-01-13T09:47:49.208142+0000 mgr.smithi084.itrhok (mgr.14180) 60 : cephadm [INF] Reconfiguring daemon mgr.smithi102.yjovaz on smithi102 2024-01-13T09:47:50.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:50 smithi084 conmon[33408]: audit 2024-01-13T09:47:49.584685+0000 mon.smithi084 (mon.0) 232 : audit [DBG] from='client.? 172.21.15.84:0/2935937377' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:47:51.251 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:47:51.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:51 smithi102 conmon[37544]: cluster 2024-01-13T09:47:50.070157+0000 mgr.smithi084.itrhok (mgr.14180) 61 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:51.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:51 smithi102 conmon[37544]: audit 2024-01-13T09:47:50. 2024-01-13T09:47:51.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:51 smithi102 conmon[37544]: 395545+0000 mon.smithi084 (mon.0) 233 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:51.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:51 smithi102 conmon[37544]: cephadm 2024-01-13T09:47:50.396089+0000 mgr.smithi084.itrhok (mgr.14180) 62 : cephadm [INF] Reconfiguring mon.smithi102 (monmap changed)... 2024-01-13T09:47:51.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:51 smithi102 conmon[37544]: audit 2024-01-13T09:47:50.396293+0000 mon.smithi084 (mon.0) 234 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-13T09:47:51.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:51 smithi102 conmon[37544]: audit 2024-01-13T09:47:50.396863+0000 mon.smithi084 (mon.0) 235 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-13T09:47:51.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:51 smithi102 conmon[37544]: audit 2024-01-13T09:47:50.397371+0000 mon.smithi084 (mon.0) 236 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:47:51.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:51 smithi102 conmon[37544]: cephadm 2024-01-13T09:47:50.397827+0000 mgr.smithi084.itrhok (mgr.14180) 63 : cephadm [INF] Reconfiguring daemon mon.smithi102 on smithi102 2024-01-13T09:47:51.584 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:51 smithi084 conmon[33408]: cluster 2024-01-13T09:47:50.070157+0000 mgr.smithi084.itrhok (mgr.14180) 61 : cluster [DBG] 2024-01-13T09:47:51.584 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:51 smithi084 conmon[33408]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:51.584 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:51 smithi084 conmon[33408]: audit 2024-01-13T09:47:50.395545+0000 mon.smithi084 (mon.0) 233 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:51.584 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:51 smithi084 conmon[33408]: cephadm 2024-01-13T09:47:50.396089+0000 mgr.smithi084.itrhok (mgr.14180) 62 : cephadm [INF] Reconfiguring mon.smithi102 (monmap changed)... 2024-01-13T09:47:51.584 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:51 smithi084 conmon[33408]: audit 2024-01-13T09:47:50.396293+0000 mon.smithi084 (mon.0) 234 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-13T09:47:51.584 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:51 smithi084 conmon[33408]: audit 2024-01-13T09:47:50.396863+0000 mon.smithi084 (mon.0) 235 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-13T09:47:51.584 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:51 smithi084 conmon[33408]: audit 2024-01-13T09:47:50.397371+0000 mon.smithi084 (mon.0) 236 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:47:51.584 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:51 smithi084 conmon[33408]: cephadm 2024-01-13T09:47:50.397827+0000 mgr.smithi084.itrhok (mgr.14180) 63 : cephadm [INF] Reconfiguring daemon mon.smithi102 on smithi102 2024-01-13T09:47:52.876 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: audit 2024-01-13T09:47:51.580744+0000 mon.smithi084 (mon.0) 237 : audit 2024-01-13T09:47:52.876 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:52.876 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: audit 2024-01-13T09:47:51.583043+0000 mon.smithi084 (mon.0) 238 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-13T09:47:52.876 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: audit 2024-01-13T09:47:51.583364+0000 mgr.smithi084.itrhok (mgr.14180) 64 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-13T09:47:52.876 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: audit 2024-01-13T09:47:51.584136+0000 mon.smithi084 (mon.0) 239 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi084:9093"}]: dispatch 2024-01-13T09:47:52.876 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: audit 2024-01-13T09:47:51. 2024-01-13T09:47:52.876 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: 584378+0000 mgr.smithi084.itrhok (mgr.14180) 65 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi084:9093"}]: dispatch 2024-01-13T09:47:52.876 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: audit 2024-01-13T09:47:51.588813 2024-01-13T09:47:52.877 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: +0000 mon.smithi084 (mon.0) 240 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:52.877 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: audit 2024-01-13T09:47: 2024-01-13T09:47:52.877 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: 51.592311+0000 mon.smithi084 (mon.0) 241 : audit 2024-01-13T09:47:52.877 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-13T09:47:52.877 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: audit 2024-01-13T09:47:52.877 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: 2024-01-13T09:47:51.592633+0000 mgr.smithi084.itrhok (mgr. 2024-01-13T09:47:52.877 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: 14180) 66 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-13T09:47:52.877 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: audit 2024-01-13T09: 2024-01-13T09:47:52.877 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: 47:51.598035+0000 mon.smithi084 (mon.0) 242 2024-01-13T09:47:52.877 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi084:3000"}]: dispatch 2024-01-13T09:47:52.877 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: audit 2024-01-13T09:47:51 2024-01-13T09:47:52.877 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: .598313+0000 mgr.smithi084.itrhok (mgr.14180) 67 : audit [DBG] 2024-01-13T09:47:52.877 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi084:3000"}]: dispatch 2024-01-13T09:47:52.877 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: audit 2024 2024-01-13T09:47:52.878 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: -01-13T09:47:51.603233+0000 mon.smithi084 (mon.0) 2024-01-13T09:47:52.878 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: 243 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:52.878 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: audit 2024-01-13T09:47: 2024-01-13T09:47:52.878 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: 51.606867+0000 mon.smithi084 (mon.0) 244 : audit 2024-01-13T09:47:52.878 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-01-13T09:47:52.878 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: audit 2024-01-13T09:47:52.878 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: 2024-01-13T09:47:51.607215+0000 mgr.smithi084.itrhok (mgr.14180 2024-01-13T09:47:52.878 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: ) 68 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-01-13T09:47:52.878 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: audit 2024-01-13T09:47 2024-01-13T09:47:52.878 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: :51.611461+0000 mon.smithi084 (mon.0) 245 : 2024-01-13T09:47:52.878 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi084:9095"}]: dispatch 2024-01-13T09:47:52.878 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: audit 2024-01-13T09:47:51. 2024-01-13T09:47:52.878 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: 611772+0000 mgr.smithi084.itrhok (mgr.14180) 69 : audit [DBG] 2024-01-13T09:47:52.879 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi084:9095"}]: dispatch 2024-01-13T09:47:52.879 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: audit 2024 2024-01-13T09:47:52.879 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: -01-13T09:47:51.616836+0000 mon.smithi084 (mon.0 2024-01-13T09:47:52.879 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: ) 246 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:52.879 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: audit 2024-01-13T09:47 2024-01-13T09:47:52.879 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: :51.618478+0000 mon.smithi084 (mon.0) 247 : audit 2024-01-13T09:47:52.879 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:52 smithi084 conmon[33408]: [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T09:47:52.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: audit 2024-01-13T09:47:51.580744+0000 2024-01-13T09:47:52.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: mon.smithi084 (mon.0) 237 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:52.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: audit 2024-01-13T09:47:51.583043+0000 mon.smithi084 (mon.0) 238 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-13T09:47:52.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: audit 2024-01-13T09:47:51.583364+0000 mgr.smithi084.itrhok (mgr.14180) 64 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-13T09:47:52.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: audit 2024-01-13T 2024-01-13T09:47:52.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: 09:47:51.584136+0000 mon.smithi084 (mon.0) 239 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi084:9093"}]: dispatch 2024-01-13T09:47:52.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: audit 2024-01-13T09:47:52.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: 2024-01-13T09:47: 2024-01-13T09:47:52.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: 51.584378+0000 mgr.smithi084.itrhok (mgr.14180) 65 : 2024-01-13T09:47:52.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi084:9093"}]: dispatch 2024-01-13T09:47:52.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: audit 2024-01-13T09:47 2024-01-13T09:47:52.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: :51.588813+0000 mon.smithi084 (mon.0) 240 : audit [INF] 2024-01-13T09:47:52.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:52.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: audit 2024-01- 2024-01-13T09:47:52.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: 13T09:47:51.592311+0000 mon.smithi084 (mon.0) 241 : audit [DBG] 2024-01-13T09:47:52.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-13T09:47:52.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: audit 2024-01-13T09:47 2024-01-13T09:47:52.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: :51.592633+0000 mgr.smithi084.itrhok (mgr.14180) 2024-01-13T09:47:52.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: 66 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-13T09:47:52.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: audit 2024-01-13T 2024-01-13T09:47:52.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: 09:47:51.598035+0000 mon.smithi084 (mon.0) 242 : audit 2024-01-13T09:47:52.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi084:3000"}]: dispatch 2024-01-13T09:47:52.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: audit 2024-01- 2024-01-13T09:47:52.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: 13T09:47:51.598313+0000 mgr.smithi084.itrhok (mgr.14180) 67 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi084:3000"}]: dispatch 2024-01-13T09:47:52.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: audit 2024-01-13T09:47:51 2024-01-13T09:47:52.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: .603233+0000 mon.smithi084 (mon.0) 243 : audit 2024-01-13T09:47:52.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:52.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: audit 2024-01-13T09:47 2024-01-13T09:47:52.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: :51.606867+0000 mon.smithi084 (mon.0) 244 : audit [DBG] 2024-01-13T09:47:52.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-01-13T09:47:52.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: audit 2024- 2024-01-13T09:47:52.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: 01-13T09:47:51.607215+0000 mgr.smithi084.itrhok (mgr.14180) 68 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-01-13T09:47:52.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: audit 2024-01- 2024-01-13T09:47:52.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: 13T09:47:51.611461+0000 mon.smithi084 (mon.0) 245 2024-01-13T09:47:52.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi084:9095"}]: dispatch 2024-01-13T09:47:52.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: audit 2024-01 2024-01-13T09:47:52.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: -13T09:47:51.611772+0000 mgr.smithi084.itrhok (mgr.14180) 69 : audit 2024-01-13T09:47:52.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi084:9095"}]: dispatch 2024-01-13T09:47:52.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: audit 2024-01 2024-01-13T09:47:52.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: -13T09:47:51.616836+0000 mon.smithi084 (mon.0) 246 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:52.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: audit 2024-01-13T09:47:51.618478+0000 2024-01-13T09:47:52.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:52 smithi102 conmon[37544]: mon.smithi084 (mon.0) 247 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T09:47:53.371 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:47:53.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:53 smithi084 conmon[33408]: cluster 2024-01-13T09:47:52.070375+0000 mgr.smithi084.itrhok (mgr.14180) 70 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:53.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:53 smithi084 conmon[33408]: audit 2024-01-13T09:47:53.013839+0000 mon.smithi084 (mon.0) 248 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:53.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:53 smithi084 conmon[33408]: audit 2024-01-13T09:47:53.018303+0000 mon.smithi084 (mon.0) 249 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:53.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:53 smithi084 conmon[33408]: audit 2024-01-13T09:47:53.381396+0000 mon.smithi084 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.84:0/595055873' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:47:53.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:53 smithi102 conmon[37544]: cluster 2024-01-13T09:47:52.070375+0000 mgr.smithi084.itrhok (mgr.14180) 70 : 2024-01-13T09:47:53.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:53 smithi102 conmon[37544]: cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:53.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:53 smithi102 conmon[37544]: audit 2024-01-13T09:47:53.013839+0000 mon.smithi084 (mon.0) 248 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:53.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:53 smithi102 conmon[37544]: audit 2024-01-13T09:47:53.018303+0000 mon.smithi084 (mon.0) 249 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:53.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:53 smithi102 conmon[37544]: audit 2024 2024-01-13T09:47:53.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:53 smithi102 conmon[37544]: -01-13T09:47:53.381396+0000 mon.smithi084 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.84:0/595055873' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:47:54.020 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2024-01-13T09:47:55.022 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:47:55.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:55 smithi102 conmon[37544]: cluster 2024-01-13T09:47:54.070568+0000 mgr.smithi084.itrhok (mgr.14180) 71 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:56.026 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:55 smithi084 conmon[33408]: cluster 2024-01-13T09:47:54.070568+0000 mgr.smithi084.itrhok (mgr.14180) 71 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:57.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:57 smithi084 conmon[33408]: audit 2024-01-13T09:47:56.036836+0000 mon.smithi084 (mon.0) 251 : audit [INF] 2024-01-13T09:47:57.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:57 smithi084 conmon[33408]: from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:57.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:57 smithi084 conmon[33408]: cluster 2024-01-13T09:47:56.070814+0000 mgr.smithi084.itrhok (mgr.14180) 72 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:57.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:57 smithi084 conmon[33408]: audit 2024-01-13T09:47:56.499318+0000 mon.smithi084 (mon.0) 252 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:57.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:57 smithi084 conmon[33408]: audit 2024-01-13T09:47:56.500074+0000 mon.smithi084 (mon.0) 253 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:47:57.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:57 smithi084 conmon[33408]: audit 2024-01-13T09:47:56.500678+0000 mon.smithi084 (mon.0) 254 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T09:47:57.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:57 smithi084 conmon[33408]: audit 2024-01-13T09:47:56.504245+0000 mon.smithi084 (mon.0) 255 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:57.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:57 smithi084 conmon[33408]: audit 2024-01-13T09:47:56.507559+0000 mon.smithi084 (mon.0) 256 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T09:47:57.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:57 smithi084 conmon[33408]: audit 2024-01-13T09:47:56.510193+0000 mon.smithi084 (mon.0) 257 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:47:57.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:57 smithi084 conmon[33408]: audit 2024-01-13T09:47:56.511035+0000 mon.smithi084 (mon.0) 258 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:47:57.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:57 smithi084 conmon[33408]: audit 2024-01-13T09:47:56.511231+0000 mon.smithi084 (mon.0) 259 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:47:57.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:57 smithi084 conmon[33408]: audit 2024-01-13T09:47:56.511669+0000 mon.smithi084 (mon.0) 260 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:47:57.304 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:47:57.374 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: audit 2024-01-13T09:47:56.036836+0000 mon.smithi084 (mon.0) 251 : audit 2024-01-13T09:47:57.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:57.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: cluster 2024-01-13T09:47:56.070814+0000 mgr.smithi084.itrhok (mgr.14180) 72 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:57.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: audit 2024-01-13T09:47:57.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: 2024-01-13T09:47:56.499318+0000 mon.smithi084 (mon.0) 252 : 2024-01-13T09:47:57.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:57.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: audit 2024 2024-01-13T09:47:57.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: -01-13T09:47:56 2024-01-13T09:47:57.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: .500074+0000 mon.smithi084 (mon.0) 2024-01-13T09:47:57.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: 253 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:47:57.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: audit 2024- 2024-01-13T09:47:57.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: 01-13T09:47:56.500678 2024-01-13T09:47:57.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: +0000 mon.smithi084 (mon.0) 254 : audit 2024-01-13T09:47:57.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T09:47:57.375 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: audit 2024-01-13T09:47:57.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: 2024-01-13T09: 2024-01-13T09:47:57.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: 47:56.504245+0000 mon.smithi084 (mon 2024-01-13T09:47:57.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: .0) 255 : audit [INF] 2024-01-13T09:47:57.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:47:57.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: audit 2024 2024-01-13T09:47:57.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: -01-13T09:47:56 2024-01-13T09:47:57.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: .507559+0000 mon.smithi084 (mon.0 2024-01-13T09:47:57.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: ) 256 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T09:47:57.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: audit 2024-01 2024-01-13T09:47:57.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: -13T09:47:56.510193+0000 2024-01-13T09:47:57.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: mon.smithi084 (mon.0) 257 : audit [INF] 2024-01-13T09:47:57.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:47:57.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: audit 2024-01 2024-01-13T09:47:57.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: -13T09:47:56.511035 2024-01-13T09:47:57.376 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: +0000 mon.smithi084 (mon.0) 258 2024-01-13T09:47:57.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:47:57.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: audit 2024-01-13T09:47:57.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: 2024-01-13T09:47 2024-01-13T09:47:57.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: :56.511231+0000 mon.smithi084 (mon. 2024-01-13T09:47:57.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: 0) 259 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:47:57.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: 2024-01-13T09:47:57.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: audit 2024-01-13 2024-01-13T09:47:57.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: T09:47:56.511669+0000 2024-01-13T09:47:57.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: mon.smithi084 (mon.0) 260 : audit 2024-01-13T09:47:57.377 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:57 smithi102 conmon[37544]: [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:47:58.070 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:58 smithi084 conmon[33408]: audit 2024-01-13T09:47:57.314187+0000 mon.smithi084 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.84:0/408352480' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:47:58.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:58 smithi102 conmon[37544]: audit 2024-01-13T09:47:57.314187+0000 mon.smithi084 ( 2024-01-13T09:47:58.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:58 smithi102 conmon[37544]: mon.0) 261 : audit [DBG] from='client.? 172.21.15.84:0/408352480' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:47:58.491 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2024-01-13T09:47:59.414 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:59 smithi102 conmon[37544]: cluster 2024-01-13T09:47:58.071052+0000 mgr.smithi084.itrhok (mgr.14180 2024-01-13T09:47:59.414 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:59 smithi102 conmon[37544]: ) 73 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:59.414 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:59 smithi102 conmon[37544]: audit 2024-01-13T09:47:59.036305+0000 mon.smithi102 (mon.1) 2 : audit [INF] from='client.? 172.21.15.102:0/1639423674' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60edc799-ec5e-47ac-833f-04ee597706f7"}]: dispatch 2024-01-13T09:47:59.414 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:47:59 smithi102 conmon[37544]: audit 2024-01-13T09:47:59.036642+0000 mon.smithi084 (mon.0) 262 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60edc799-ec5e-47ac-833f-04ee597706f7"}]: dispatch 2024-01-13T09:47:59.493 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:47:59.524 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:59 smithi084 conmon[33408]: cluster 2024-01-13T09:47:58.071052+0000 mgr.smithi084.itrhok (mgr.14180) 73 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:47:59.524 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:59 smithi084 conmon[33408]: audit 2024-01-13T09:47:59.036305+0000 mon.smithi102 (mon.1) 2 : audit [INF] from='client.? 172.21.15.102:0/1639423674' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60edc799-ec5e-47ac-833f-04ee597706f7"}]: dispatch 2024-01-13T09:47:59.524 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:47:59 smithi084 conmon[33408]: audit 2024-01-13T09:47:59.036642+0000 mon.smithi084 (mon.0) 262 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60edc799-ec5e-47ac-833f-04ee597706f7"}]: dispatch 2024-01-13T09:48:00.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:00 smithi102 conmon[37544]: audit 2024-01-13T09:47:59.041819+0000 mon.smithi084 (mon.0 2024-01-13T09:48:00.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:00 smithi102 conmon[37544]: ) 263 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60edc799-ec5e-47ac-833f-04ee597706f7"}]': finished 2024-01-13T09:48:00.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:00 smithi102 conmon[37544]: cluster 2024-01-13T09:47:59.041863+0000 mon.smithi084 (mon.0) 264 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-01-13T09:48:00.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:00 smithi102 conmon[37544]: audit 2024-01-13T09:47:59.042290+0000 mon.smithi084 (mon.0) 265 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:48:00.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:00 smithi102 conmon[37544]: audit 2024-01-13T09:47:59.325411+0000 mon.smithi102 (mon.1) 3 : audit [INF] from='client.? 172.21.15.102:0/1438218352' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:48:00.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:00 smithi102 conmon[37544]: audit 2024-01-13T09:47:59.325735+0000 mon.smithi084 (mon.0) 266 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:48:00.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:00 smithi102 conmon[37544]: audit 2024-01- 2024-01-13T09:48:00.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:00 smithi102 conmon[37544]: 13T09:47:59.335787+0000 mon.smithi084 (mon.0) 267 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:48:00.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:00 smithi102 conmon[37544]: cluster 2024-01-13T09:47:59.335832+0000 mon.smithi084 (mon.0) 268 : cluster [DBG] osdmap e7: 0 total, 0 up, 0 in 2024-01-13T09:48:00.235 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:00 smithi102 conmon[37544]: audit 2024-01 2024-01-13T09:48:00.235 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:00 smithi102 conmon[37544]: -13T09:47:59.848698+0000 mon.smithi084 (mon.0) 269 : audit [INF] from='client.? 172.21.15.84:0/197586998' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c7160343-b382-47fc-a49a-84330b559470"}]: dispatch 2024-01-13T09:48:00.235 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:00 smithi102 conmon[37544]: audit 2024- 2024-01-13T09:48:00.235 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:00 smithi102 conmon[37544]: 01-13T09:47:59.855641+0000 mon.smithi084 (mon.0) 270 : audit [INF] from='client.? 172.21.15.84:0/197586998' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c7160343-b382-47fc-a49a-84330b559470"}]': finished 2024-01-13T09:48:00.235 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:00 smithi102 conmon[37544]: cluster 2024-01-13T09:47:59.855678+0000 mon.smithi084 (mon.0) 271 2024-01-13T09:48:00.235 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:00 smithi102 conmon[37544]: : cluster [DBG] osdmap e8: 1 total, 0 up, 1 in 2024-01-13T09:48:00.235 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:00 smithi102 conmon[37544]: audit 2024-01 2024-01-13T09:48:00.235 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:00 smithi102 conmon[37544]: -13T09:47:59.855786+0000 mon.smithi084 (mon.0) 272 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:48:00.273 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:00 smithi084 conmon[33408]: audit 2024-01-13T09:47:59.041819+0000 mon.smithi084 (mon.0) 2024-01-13T09:48:00.274 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:00 smithi084 conmon[33408]: 263 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60edc799-ec5e-47ac-833f-04ee597706f7"}]': finished 2024-01-13T09:48:00.274 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:00 smithi084 conmon[33408]: cluster 2024-01-13T09:47:59.041863+0000 mon.smithi084 (mon.0) 264 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-01-13T09:48:00.274 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:00 smithi084 conmon[33408]: audit 2024-01-13T09:47:59.042290+0000 mon.smithi084 (mon.0) 2024-01-13T09:48:00.274 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:00 smithi084 conmon[33408]: 265 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:48:00.274 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:00 smithi084 conmon[33408]: audit 2024-01-13T09:47 2024-01-13T09:48:00.274 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:00 smithi084 conmon[33408]: :59.325411+0000 mon.smithi102 (mon.1) 3 : audit 2024-01-13T09:48:00.274 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:00 smithi084 conmon[33408]: [INF] from='client.? 172.21.15.102:0/1438218352' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:48:00.274 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:00 smithi084 conmon[33408]: audit 2024-01-13T09:47 2024-01-13T09:48:00.274 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:00 smithi084 conmon[33408]: :59.325735+0000 mon.smithi084 (mon.0) 266 : audit 2024-01-13T09:48:00.274 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:00 smithi084 conmon[33408]: [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:48:00.274 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:00 smithi084 conmon[33408]: audit 2024-01-13T09:47: 2024-01-13T09:48:00.274 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:00 smithi084 conmon[33408]: 59.335787+0000 mon.smithi084 (mon.0) 267 : audit 2024-01-13T09:48:00.275 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:00 smithi084 conmon[33408]: [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:48:00.275 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:00 smithi084 conmon[33408]: cluster 2024-01-13T09:47 2024-01-13T09:48:00.275 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:00 smithi084 conmon[33408]: :59.335832+0000 mon.smithi084 (mon.0) 268 : 2024-01-13T09:48:00.275 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:00 smithi084 conmon[33408]: cluster [DBG] osdmap e7: 0 total, 0 up, 0 in 2024-01-13T09:48:00.275 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:00 smithi084 conmon[33408]: audit 2024-01-13T09:47 2024-01-13T09:48:00.275 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:00 smithi084 conmon[33408]: :59.848698+0000 mon.smithi084 (mon.0) 269 : 2024-01-13T09:48:00.275 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:00 smithi084 conmon[33408]: audit [INF] from='client.? 172.21.15.84:0/197586998' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c7160343-b382-47fc-a49a-84330b559470"}]: dispatch 2024-01-13T09:48:00.275 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:00 smithi084 conmon[33408]: audit 2024-01-13T09: 2024-01-13T09:48:00.275 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:00 smithi084 conmon[33408]: 47:59.855641+0000 mon.smithi084 (mon.0) 270 2024-01-13T09:48:00.275 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:00 smithi084 conmon[33408]: : audit [INF] from='client.? 172.21.15.84:0/197586998' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c7160343-b382-47fc-a49a-84330b559470"}]': finished 2024-01-13T09:48:00.275 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:00 smithi084 conmon[33408]: cluster 2024-01-13T09:47 2024-01-13T09:48:00.275 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:00 smithi084 conmon[33408]: :59.855678+0000 mon.smithi084 (mon.0) 271 : cluster 2024-01-13T09:48:00.275 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:00 smithi084 conmon[33408]: [DBG] osdmap e8: 1 total, 0 up, 1 in 2024-01-13T09:48:00.275 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:00 smithi084 conmon[33408]: audit 2024-01-13T09:47: 2024-01-13T09:48:00.275 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:00 smithi084 conmon[33408]: 59.855786+0000 mon.smithi084 (mon.0) 272 : audit 2024-01-13T09:48:00.276 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:00 smithi084 conmon[33408]: [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:48:01.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: cephadm 2024-01-13T09:48:00.006499+0000 mgr.smithi084.itrhok (mgr.14180) 74 : cephadm [ERR] 2024-01-13T09:48:01.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: executing create_from_spec_one(([('smithi084', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi102', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:48:01.485 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:48:01.485 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:48:01.485 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: return f(*arg) 2024-01-13T09:48:01.485 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:48:01.485 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:48:01.485 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:48:01.485 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: code, '\n'.join(err))) 2024-01-13T09:48:01.485 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:48:01.485 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmprf9thix9:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpnb8aa8bv:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:48:01.485 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:48:01.485 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:48:01.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:48:01.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 60edc799-ec5e-47ac-833f-04ee597706f7 2024-01-13T09:48:01.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:48:01.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:48:01.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:48:01.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:48:01.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:48:01.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr self.prepare() 2024-01-13T09:48:01.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:01.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:01.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:48:01.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:48:01.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:48:01.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:48:01.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:48:01.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:48:01.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:48:01.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:48:01.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:48:01.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:48:01.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:48:01.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:48:01.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:48:01.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:48:01.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:48:01.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:48:01.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:48:01.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:48:01.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:48:01.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:48:01.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:48:01.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:48:01.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:48:01.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:48:01.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:48:01.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:01.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:01.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:48:01.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:48:01.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:48:01.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:48:01.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:01.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:01.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:48:01.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:48:01.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:48:01.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:48:01.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:48:01.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:48:01.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:48:01.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:48:01.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:01.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:01.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:48:01.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:48:01.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:48:01.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:48:01.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:48:01.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:48:01.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:48:01.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: main() 2024-01-13T09:48:01.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:48:01.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: r = ctx.func(ctx) 2024-01-13T09:48:01.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:48:01.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:48:01.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:48:01.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:48:01.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:48:01.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:48:01.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927 2024-01-13T09:48:01.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: , in _validate_fsid 2024-01-13T09:48:01.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:48:01.492 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:48:01.492 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:48:01.492 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:48:01.492 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:48:01.492 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmprf9thix9:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpnb8aa8bv:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:48:01.492 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: cluster 2024-01-13T09:48:00.071234+0000 mgr.smithi084.itrhok (mgr.14180) 75 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:01.492 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: audit 2024-01-13T09:48:00.158605+0000 mon.smithi084 (mon.0) 273 : audit [INF] from='client.? 172.21.15.84:0/3106993806' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:48:01.492 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: audit 2024-01-13T09:48:00.162846+0000 mon.smithi084 (mon.0) 274 : audit [INF] from='client.? 172.21.15.84:0/3106993806' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:48:01.492 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:01 smithi102 conmon[37544]: cluster 2024-01-13T09:48:00.162884+0000 mon.smithi084 (mon.0) 275 : cluster [DBG] osdmap e9: 0 total, 0 up, 0 in 2024-01-13T09:48:01.500 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: cephadm 2024-01-13T09:48:00.006499+0000 mgr.smithi084.itrhok (mgr.14180) 74 : cephadm [ERR] executing create_from_spec_one(([('smithi084', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi102', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:48:01.503 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:48:01.503 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:48:01.503 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: return f(*arg) 2024-01-13T09:48:01.503 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:48:01.503 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:48:01.503 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:48:01.503 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: code, '\n'.join(err))) 2024-01-13T09:48:01.503 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:48:01.504 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmprf9thix9:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpnb8aa8bv:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:48:01.504 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:48:01.504 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:48:01.504 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:48:01.504 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 60edc799-ec5e-47ac-833f-04ee597706f7 2024-01-13T09:48:01.504 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:48:01.504 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:48:01.504 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:48:01.504 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:48:01.504 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:48:01.504 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr self.prepare() 2024-01-13T09:48:01.504 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:01.505 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:01.505 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:48:01.505 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:48:01.505 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:48:01.505 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:48:01.505 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:48:01.505 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:48:01.505 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:48:01.505 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:48:01.505 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:48:01.505 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:48:01.505 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:48:01.506 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:48:01.506 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:48:01.506 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:48:01.506 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:48:01.506 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:48:01.506 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:48:01.506 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:48:01.506 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:48:01.506 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:48:01.506 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:48:01.506 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:48:01.506 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:48:01.506 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:48:01.507 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:48:01.507 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:01.507 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:01.507 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:48:01.507 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:48:01.507 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:48:01.507 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:48:01.507 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:01.507 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:01.507 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:48:01.507 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:48:01.507 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:48:01.508 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:48:01.508 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:48:01.508 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:48:01.508 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:48:01.508 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:48:01.508 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:01.508 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:01.508 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:48:01.508 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:48:01.508 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:48:01.508 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:48:01.509 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:48:01.509 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:48:01.509 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:48:01.509 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: main() 2024-01-13T09:48:01.509 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:48:01.509 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: r = ctx.func(ctx) 2024-01-13T09:48:01.509 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:48:01.509 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:48:01.509 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:48:01.509 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:48:01.509 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:48:01.509 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:48:01.509 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774 2024-01-13T09:48:01.510 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:48:01.510 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:48:01.510 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:48:01.510 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:48:01.510 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:48:01.510 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:48:01.510 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmprf9thix9:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpnb8aa8bv:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:48:01.510 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: cluster 2024-01-13T09:48:00.071234+0000 mgr.smithi084.itrhok (mgr.14180) 75 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:01.510 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: audit 2024-01-13T09:48:00.158605+0000 mon.smithi084 (mon.0) 273 : audit [INF] from='client.? 172.21.15.84:0/3106993806' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:48:01.510 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: audit 2024-01-13T09:48:00.162846+0000 mon.smithi084 (mon.0) 274 : audit [INF] from='client.? 172.21.15.84:0/3106993806' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:48:01.510 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:01 smithi084 conmon[33408]: cluster 2024-01-13T09:48:00.162884+0000 mon.smithi084 (mon.0) 275 : cluster [DBG] osdmap e9: 0 total, 0 up, 0 in 2024-01-13T09:48:01.891 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:48:02.278 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: cephadm 2024-01-13T09:48:01.211485+0000 mgr.smithi084.itrhok (mgr.14180) 76 : cephadm [ERR] executing create_from_spec_one(([('smithi084', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi102', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:48:02.278 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:48:02.278 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:48:02.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: return f(*arg) 2024-01-13T09:48:02.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:48:02.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:48:02.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:48:02.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: code, '\n'.join(err))) 2024-01-13T09:48:02.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi084/config 2024-01-13T09:48:02.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpd_s78vqs:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp6o0lxu5e:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:48:02.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:48:02.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:48:02.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:48:02.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new c7160343-b382-47fc-a49a-84330b559470 2024-01-13T09:48:02.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:48:02.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:48:02.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:48:02.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:48:02.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:48:02.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr self.prepare() 2024-01-13T09:48:02.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:02.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:02.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:48:02.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:48:02.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:48:02.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:48:02.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:48:02.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:48:02.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:48:02.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:48:02.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:48:02.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:48:02.281 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:48:02.281 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:48:02.281 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:48:02.281 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:48:02.281 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:48:02.281 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:48:02.281 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:48:02.281 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:48:02.281 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:48:02.281 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:48:02.281 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:48:02.281 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:48:02.281 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:48:02.281 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:48:02.281 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:48:02.282 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:02.282 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:02.282 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:48:02.282 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:48:02.282 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:48:02.282 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:48:02.282 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:02.282 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:02.282 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:48:02.282 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:48:02.282 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:48:02.282 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:48:02.282 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:48:02.282 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:48:02.282 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:48:02.283 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:48:02.283 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:02.283 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:02.283 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:48:02.283 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:48:02.283 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:48:02.283 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:48:02.283 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:48:02.283 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:48:02.283 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:48:02.283 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: main() 2024-01-13T09:48:02.283 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:48:02.283 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: r = ctx.func(ctx) 2024-01-13T09:48:02.283 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:48:02.283 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:48:02.284 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:48:02.284 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:48:02.284 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:48:02.284 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:48:02.284 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774 2024-01-13T09:48:02.284 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:48:02.284 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:48:02.284 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:48:02.284 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:48:02.284 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:48:02.284 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:48:02.284 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpd_s78vqs:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp6o0lxu5e:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:48:02.284 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: cephadm 2024-01-13T09:48:01.213103+0000 mgr.smithi084.itrhok (mgr.14180) 77 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T09:48:02.284 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: service_id: all-available-devices 2024-01-13T09:48:02.285 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: service_name: osd.all-available-devices 2024-01-13T09:48:02.285 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: placement: 2024-01-13T09:48:02.285 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: host_pattern: '*' 2024-01-13T09:48:02.285 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: spec: 2024-01-13T09:48:02.285 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: data_devices: 2024-01-13T09:48:02.285 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: all: true 2024-01-13T09:48:02.285 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: filter_logic: AND 2024-01-13T09:48:02.285 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: objectstore: bluestore 2024-01-13T09:48:02.285 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:48:02.285 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmprf9thix9:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpnb8aa8bv:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:48:02.285 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:48:02.285 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:48:02.285 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:48:02.285 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 60edc799-ec5e-47ac-833f-04ee597706f7 2024-01-13T09:48:02.286 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:48:02.286 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:48:02.286 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:48:02.286 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:48:02.286 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:48:02.286 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr self.prepare() 2024-01-13T09:48:02.286 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:02.286 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:02.286 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:48:02.286 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:48:02.286 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:48:02.286 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:48:02.286 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:48:02.286 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:48:02.286 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:48:02.287 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:48:02.287 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:48:02.287 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:48:02.287 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:48:02.287 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:48:02.287 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:48:02.287 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:48:02.287 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:48:02.287 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:48:02.287 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:48:02.287 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:48:02.287 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:48:02.287 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:48:02.287 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:48:02.288 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:48:02.288 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:48:02.288 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:48:02.288 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:48:02.288 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:02.288 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:02.288 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:48:02.288 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:48:02.288 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:48:02.288 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:48:02.288 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:02.288 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:02.288 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:48:02.288 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:48:02.289 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:48:02.289 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:48:02.289 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:48:02.289 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:48:02.289 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:48:02.289 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:48:02.289 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:02.289 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:02.289 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lv 2024-01-13T09:48:02.289 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: m/zap.py", line 302, in zap_osd 2024-01-13T09:48:02.289 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:48:02.289 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:48:02.289 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:48:02.289 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:48:02.289 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:48:02.290 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:48:02.290 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: main() 2024-01-13T09:48:02.290 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:48:02.290 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: r = ctx.func(ctx) 2024-01-13T09:48:02.290 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:48:02.290 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:48:02.290 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:48:02.290 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:48:02.290 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:48:02.290 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:48:02.290 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:48:02.290 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:48:02.290 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:48:02.290 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:48:02.290 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:48:02.290 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:48:02.291 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmprf9thix9:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpnb8aa8bv:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:48:02.291 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:48:02.291 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T09:48:02.291 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: if self._apply_service(spec): 2024-01-13T09:48:02.291 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T09:48:02.291 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T09:48:02.291 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T09:48:02.291 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T09:48:02.291 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T09:48:02.291 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T09:48:02.291 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T09:48:02.291 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T09:48:02.291 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T09:48:02.291 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: raise self._value 2024-01-13T09:48:02.291 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T09:48:02.292 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: result = (True, func(*args, **kwds)) 2024-01-13T09:48:02.292 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T09:48:02.292 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: return list(map(*args)) 2024-01-13T09:48:02.292 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:48:02.292 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: return f(*arg) 2024-01-13T09:48:02.292 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:48:02.292 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:48:02.292 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:48:02.292 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: code, '\n'.join(err))) 2024-01-13T09:48:02.292 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:48:02.292 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmprf9thix9:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpnb8aa8bv:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:48:02.292 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:48:02.292 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:48:02.292 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:48:02.293 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 60edc799-ec5e-47ac-833f-04ee597706f7 2024-01-13T09:48:02.293 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:48:02.293 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:48:02.293 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:48:02.293 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:48:02.293 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:48:02.293 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr self.prepare() 2024-01-13T09:48:02.293 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:02.293 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:02.293 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:48:02.293 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:48:02.293 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:48:02.293 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:48:02.293 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:48:02.293 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:48:02.293 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:48:02.294 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:48:02.294 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:48:02.294 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:48:02.294 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:48:02.294 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:48:02.294 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:48:02.294 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:48:02.294 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:48:02.294 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr F 2024-01-13T09:48:02.294 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: ile "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:48:02.294 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:48:02.294 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:48:02.294 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:48:02.294 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:48:02.294 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:48:02.295 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:48:02.295 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:48:02.295 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:48:02.295 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:48:02.295 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:02.295 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:02.295 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:48:02.295 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:48:02.295 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:48:02.295 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:48:02.295 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:02.295 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:02.295 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:48:02.295 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:48:02.295 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:48:02.295 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:48:02.296 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:48:02.296 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:48:02.296 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:48:02.296 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:48:02.297 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:02.297 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:02.297 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:48:02.297 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:48:02.297 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:48:02.297 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:48:02.297 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:48:02.297 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:48:02.297 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:48:02.297 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: main() 2024-01-13T09:48:02.297 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:48:02.297 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: r = ctx.func(ctx) 2024-01-13T09:48:02.297 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:48:02.298 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:48:02.298 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:48:02.298 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:48:02.298 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:48:02.298 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:48:02.298 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:48:02.298 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:48:02.298 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:48:02.298 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:48:02.298 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:48:02.298 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:48:02.298 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmprf9thix9:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpnb8aa8bv:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:48:02.298 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: cluster 2024-01-13T09:48:01.213650+0000 mgr.smithi084.itrhok (mgr.14180) 78 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:02.298 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: audit 2024-01-13T09:48:01.214371+0000 mon.smithi084 (mon.0) 276 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T09:48:02.299 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:02 smithi084 conmon[33408]: audit 2024-01-13T09:48:01.900639+0000 mon.smithi084 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.84:0/14558286' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:48:02.485 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: cephadm 2024-01-13T09:48:01.211485+0000 mgr.smithi084.itrhok (mgr.14180) 76 : cephadm [ERR] 2024-01-13T09:48:02.485 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: executing create_from_spec_one(([('smithi084', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi102', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:48:02.485 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:48:02.485 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:48:02.485 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: return f(*arg) 2024-01-13T09:48:02.485 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:48:02.485 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:48:02.485 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:48:02.485 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: code, '\n'.join(err))) 2024-01-13T09:48:02.485 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi084/config 2024-01-13T09:48:02.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpd_s78vqs:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp6o0lxu5e:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:48:02.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:48:02.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:48:02.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:48:02.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new c7160343-b382-47fc-a49a-84330b559470 2024-01-13T09:48:02.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:48:02.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:48:02.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:48:02.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:48:02.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:48:02.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr self.prepare() 2024-01-13T09:48:02.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:02.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:02.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:48:02.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:48:02.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:48:02.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:48:02.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:48:02.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:48:02.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:48:02.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:48:02.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:48:02.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:48:02.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:48:02.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:48:02.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:48:02.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:48:02.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:48:02.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:48:02.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:48:02.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:48:02.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:48:02.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:48:02.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:48:02.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:48:02.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:48:02.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:48:02.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:48:02.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:02.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:02.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:48:02.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:48:02.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:48:02.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:48:02.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:02.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:02.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:48:02.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:48:02.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:48:02.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:48:02.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:48:02.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:48:02.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:48:02.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:48:02.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:02.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:02.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:48:02.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:48:02.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:48:02.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:48:02.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:48:02.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:48:02.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:48:02.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: main() 2024-01-13T09:48:02.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:48:02.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: r = ctx.func(ctx) 2024-01-13T09:48:02.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:48:02.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:48:02.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:48:02.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:48:02.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:48:02.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:48:02.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 192 2024-01-13T09:48:02.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: 7, in _validate_fsid 2024-01-13T09:48:02.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:48:02.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:48:02.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:48:02.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:48:02.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:48:02.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpd_s78vqs:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp6o0lxu5e:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:48:02.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: cephadm 2024-01-13T09:48:01.213103+0000 mgr.smithi084.itrhok (mgr.14180) 77 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T09:48:02.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: service_id: all-available-devices 2024-01-13T09:48:02.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: service_name: osd.all-available-devices 2024-01-13T09:48:02.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: placement: 2024-01-13T09:48:02.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: host_pattern: '*' 2024-01-13T09:48:02.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: spec: 2024-01-13T09:48:02.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: data_devices: 2024-01-13T09:48:02.492 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: all: true 2024-01-13T09:48:02.492 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: filter_logic: AND 2024-01-13T09:48:02.492 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: objectstore: bluestore 2024-01-13T09:48:02.492 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:48:02.492 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmprf9thix9:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpnb8aa8bv:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:48:02.492 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:48:02.492 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:48:02.492 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:48:02.492 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 60edc799-ec5e-47ac-833f-04ee597706f7 2024-01-13T09:48:02.492 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:48:02.492 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:48:02.492 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:48:02.492 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:48:02.493 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:48:02.493 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr self.prepare() 2024-01-13T09:48:02.493 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:02.493 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:02.493 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:48:02.493 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:48:02.493 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:48:02.493 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:48:02.493 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:48:02.493 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:48:02.493 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:48:02.493 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:48:02.493 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:48:02.493 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:48:02.494 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:48:02.494 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:48:02.494 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:48:02.494 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:48:02.494 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:48:02.494 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:48:02.494 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:48:02.494 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:48:02.494 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:48:02.494 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:48:02.494 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:48:02.494 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:48:02.494 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:48:02.494 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:48:02.495 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:48:02.495 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:02.496 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:02.496 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:48:02.496 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:48:02.496 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:48:02.496 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:48:02.496 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:02.496 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:02.496 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:48:02.496 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:48:02.496 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:48:02.496 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:48:02.496 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:48:02.496 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:48:02.496 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:48:02.496 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:48:02.497 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:02.497 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:02.497 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:48:02.497 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr devices = find_associated_devices(s 2024-01-13T09:48:02.497 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: elf.args.osd_id, self.args.osd_fsid) 2024-01-13T09:48:02.497 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:48:02.497 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:48:02.497 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:48:02.497 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:48:02.497 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:48:02.497 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: main() 2024-01-13T09:48:02.497 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:48:02.497 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: r = ctx.func(ctx) 2024-01-13T09:48:02.497 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:48:02.498 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:48:02.498 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:48:02.498 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:48:02.498 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:48:02.498 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:48:02.498 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:48:02.498 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:48:02.498 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:48:02.498 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:48:02.498 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:48:02.498 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:48:02.498 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmprf9thix9:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpnb8aa8bv:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:48:02.498 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:48:02.499 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T09:48:02.499 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: if self._apply_service(spec): 2024-01-13T09:48:02.499 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T09:48:02.499 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T09:48:02.499 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T09:48:02.499 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T09:48:02.499 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T09:48:02.499 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T09:48:02.499 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T09:48:02.499 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T09:48:02.499 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T09:48:02.499 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: raise self._value 2024-01-13T09:48:02.499 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T09:48:02.499 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: result = (True, func(*args, **kwds)) 2024-01-13T09:48:02.499 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T09:48:02.499 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: return list(map(*args)) 2024-01-13T09:48:02.500 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:48:02.500 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: return f(*arg) 2024-01-13T09:48:02.500 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:48:02.500 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:48:02.500 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:48:02.500 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: code, '\n'.join(err))) 2024-01-13T09:48:02.500 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:48:02.500 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmprf9thix9:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpnb8aa8bv:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:48:02.500 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:48:02.500 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:48:02.500 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:48:02.500 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 60edc799-ec5e-47ac-833f-04ee597706f7 2024-01-13T09:48:02.500 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:48:02.500 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:48:02.501 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:48:02.501 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:48:02.501 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:48:02.501 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr self.prepare() 2024-01-13T09:48:02.501 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:02.501 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:02.501 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:48:02.501 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:48:02.501 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:48:02.501 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:48:02.501 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:48:02.501 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:48:02.501 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:48:02.501 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:48:02.501 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:48:02.502 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:48:02.502 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:48:02.502 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:48:02.502 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:48:02.502 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:48:02.502 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:48:02.502 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:48:02.502 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/ 2024-01-13T09:48:02.502 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: podman: stderr return f(*a, **kw) 2024-01-13T09:48:02.502 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:48:02.502 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:48:02.502 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:48:02.502 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:48:02.502 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:48:02.502 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:48:02.502 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:48:02.503 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:48:02.503 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:02.503 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:02.503 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:48:02.503 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:48:02.503 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:48:02.503 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:48:02.503 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:02.503 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:02.503 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:48:02.503 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:48:02.503 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:48:02.503 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:48:02.503 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:48:02.503 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:48:02.503 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:48:02.504 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:48:02.504 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:02.504 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:02.504 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:48:02.504 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:48:02.504 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:48:02.504 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:48:02.504 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:48:02.504 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:48:02.504 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:48:02.504 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: main() 2024-01-13T09:48:02.504 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:48:02.504 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: r = ctx.func(ctx) 2024-01-13T09:48:02.504 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:48:02.504 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:48:02.505 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:48:02.505 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:48:02.505 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:48:02.505 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:48:02.505 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:48:02.505 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:48:02.505 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:48:02.505 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:48:02.505 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:48:02.505 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:48:02.505 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmprf9thix9:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpnb8aa8bv:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:48:02.505 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: cluster 2024-01-13T09:48:01.213650+0000 mgr.smithi084.itrhok (mgr.14180) 78 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:02.505 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: audit 2024-01-13T09:48:01.214371+0000 mon.smithi084 (mon.0) 276 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T09:48:02.505 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:02 smithi102 conmon[37544]: audit 2024-01-13T09:48:01.900639+0000 mon.smithi084 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.84:0/14558286' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:48:02.825 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":9,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:48:03.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:03 smithi084 conmon[33408]: cluster 2024-01-13T09:48:02.056876+0000 mon.smithi084 (mon 2024-01-13T09:48:03.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:03 smithi084 conmon[33408]: .0) 278 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T09:48:03.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:03 smithi084 conmon[33408]: audit 2024-01-13T09:48:02.595140+0000 mon.smithi084 (mon.0) 279 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:48:03.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:03 smithi102 conmon[37544]: cluster 2024-01-13T09:48:02.056876+0000 mon.smithi084 (mon.0) 278 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T09:48:03.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:03 smithi102 conmon[37544]: audit 2024-01-13T09:48:02.595140+0000 mon.smithi084 (mon.0) 279 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:48:03.826 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:48:04.276 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:04 smithi084 conmon[33408]: cluster 2024-01-13T09:48:03.214013+0000 mgr.smithi084.itrhok (mgr.14180) 2024-01-13T09:48:04.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:04 smithi084 conmon[33408]: 79 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:04.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:04 smithi102 conmon[37544]: cluster 2024-01-13T09:48:03.214013+0000 mgr.smithi084.itrhok (mgr.14180) 2024-01-13T09:48:04.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:04 smithi102 conmon[37544]: 79 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:06.110 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:05 smithi102 conmon[37544]: audit 2024-01-13T09:48:04.942761+0000 mon.smithi084 (mon.0) 280 : audit 2024-01-13T09:48:06.110 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:05 smithi102 conmon[37544]: [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:48:06.110 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:05 smithi102 conmon[37544]: audit 2024-01-13T09:48:04.943502+0000 mon.smithi084 (mon.0) 281 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:48:06.110 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:05 smithi102 conmon[37544]: audit 2024-01-13T09:48:04.944107+0000 mon.smithi084 (mon.0) 282 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T09:48:06.110 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:05 smithi102 conmon[37544]: audit 2024-01-13T09:48:04.950737+0000 mon.smithi084 (mon.0) 283 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:48:06.110 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:05 smithi102 conmon[37544]: cluster 2024-01-13T09:48: 2024-01-13T09:48:06.110 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:05 smithi102 conmon[37544]: 04.951024+0000 mgr.smithi084.itrhok (mgr.14180) 80 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:06.110 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:05 smithi102 conmon[37544]: audit 2024-01-13T09:48:04.954052+0000 mon.smithi084 (mon.0) 284 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T09:48:06.110 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:05 smithi102 conmon[37544]: audit 2024-01-13T09:48:04.956737+0000 mon.smithi084 (mon.0) 285 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:48:06.110 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:05 smithi102 conmon[37544]: audit 2024-01-13T09:48:04.957622+0000 2024-01-13T09:48:06.110 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:05 smithi102 conmon[37544]: mon.smithi084 (mon.0) 286 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:48:06.111 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:05 smithi102 conmon[37544]: audit 2024-01-13T09:48:04.957838+0000 mon.smithi084 (mon.0 2024-01-13T09:48:06.111 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:05 smithi102 conmon[37544]: ) 287 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:48:06.111 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:05 smithi102 conmon[37544]: audit 2024 2024-01-13T09:48:06.111 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:05 smithi102 conmon[37544]: -01-13T09:48:04.958267+0000 mon.smithi084 (mon.0) 288 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:48:06.154 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:05 smithi084 conmon[33408]: audit 2024-01-13T09:48:04.942761+0000 mon.smithi084 (mon.0) 280 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:48:06.155 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:05 smithi084 conmon[33408]: audit 2024-01-13T09:48:04.943502+0000 mon.smithi084 (mon.0) 281 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:48:06.155 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:05 smithi084 conmon[33408]: audit 2024-01-13T09:48:04.944107+0000 mon.smithi084 (mon.0) 282 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T09:48:06.155 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:05 smithi084 conmon[33408]: audit 2024-01-13T09:48:04.950737+0000 mon.smithi084 (mon.0) 283 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:48:06.155 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:05 smithi084 conmon[33408]: cluster 2024-01-13T09:48:04.951024+0000 mgr.smithi084.itrhok (mgr.14180) 80 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:06.155 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:05 smithi084 conmon[33408]: audit 2024-01-13T09:48:04.954052+0000 mon.smithi084 (mon.0) 284 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T09:48:06.155 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:05 smithi084 conmon[33408]: audit 2024-01-13T09:48:04.956737+0000 mon.smithi084 (mon.0) 285 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:48:06.155 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:05 smithi084 conmon[33408]: audit 2024-01-13T09:48:04.957622+0000 mon.smithi084 (mon.0) 286 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:48:06.155 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:05 smithi084 conmon[33408]: audit 2024-01-13T09:48:06.155 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:05 smithi084 conmon[33408]: 2024-01-13T09:48:04.957838+0000 mon.smithi084 (mon.0) 287 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:48:06.155 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:05 smithi084 conmon[33408]: audit 2024-01-13T09:48:04.958267+0000 mon.smithi084 (mon.0) 288 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:48:06.267 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:48:07.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:06 smithi102 conmon[37544]: cluster 2024-01-13T09:48:05.949550+0000 mon.smithi084 (mon.0) 289 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T09:48:07.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:06 smithi102 conmon[37544]: cluster 2024-01-13T09:48:05.949599+0000 mon.smithi084 (mon.0) 290 : cluster [INF] Cluster is now healthy 2024-01-13T09:48:07.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:06 smithi102 conmon[37544]: audit 2024-01-13T09:48:06.277571+0000 mon.smithi084 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.84:0/1792318455' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:48:07.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:06 smithi084 conmon[33408]: cluster 2024-01-13T09:48:05.949550+0000 mon.smithi084 (mon.0) 289 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T09:48:07.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:06 smithi084 conmon[33408]: 2024-01-13T09:48:07.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:06 smithi084 conmon[33408]: cluster 2024-01-13T09:48:05.949599+0000 mon.smithi084 (mon.0) 290 : cluster [INF] Cluster is now healthy 2024-01-13T09:48:07.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:06 smithi084 conmon[33408]: audit 2024-01-13T09:48:06.277571+0000 mon.smithi084 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.84:0/1792318455' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:48:07.417 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":9,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:48:08.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:07 smithi102 conmon[37544]: cluster 2024-01-13T09:48:06.951327+0000 mgr.smithi084.itrhok (mgr.14180) 81 2024-01-13T09:48:08.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:07 smithi102 conmon[37544]: : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:08.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:07 smithi102 conmon[37544]: audit 2024-01-13T09:48:07.536870+0000 mon.smithi102 (mon.1) 4 : audit [INF] from='client.? 172.21.15.102:0/295376378' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d524ee64-298b-4417-9902-98f78ab71a6a"}]: dispatch 2024-01-13T09:48:08.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:07 smithi102 conmon[37544]: audit 2024-01-13T09:48:07.537312+0000 mon.smithi084 (mon.0) 292 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d524ee64-298b-4417-9902-98f78ab71a6a"}]: dispatch 2024-01-13T09:48:08.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:07 smithi102 conmon[37544]: audit 2024-01-13T09:48:07.540670+0000 mon.smithi084 (mon.0) 293 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d524ee64-298b-4417-9902-98f78ab71a6a"}]': finished 2024-01-13T09:48:08.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:07 smithi102 conmon[37544]: cluster 2024-01-13T09:48:07.540717+0000 mon.smithi084 (mon.0) 294 : cluster [DBG] osdmap e10: 1 total, 0 up, 1 in 2024-01-13T09:48:08.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:07 smithi102 conmon[37544]: audit 2024-01-13T09:48:07.540807+0000 mon.smithi084 (mon.0) 295 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:48:08.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:07 smithi102 conmon[37544]: audit 2024-01-13T09:48:07.831029+0000 mon.smithi102 (mon.1) 5 : audit [INF] from='client.? 172.21.15.102:0/231390807' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:48:08.235 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:07 smithi102 conmon[37544]: audit 2024-01-13T09:48:07.831487+0000 mon.smithi084 (mon.0) 296 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:48:08.235 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:07 smithi102 conmon[37544]: audit 2024-01-13T09:48:07.835078+0000 mon.smithi084 (mon.0) 297 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:48:08.235 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:07 smithi102 conmon[37544]: cluster 2024-01-13T09:48:07.835148+0000 mon.smithi084 (mon.0) 298 : cluster [DBG] osdmap e11: 0 total, 0 up, 0 in 2024-01-13T09:48:08.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:07 smithi084 conmon[33408]: cluster 2024-01-13T09:48:06.951327+0000 mgr.smithi084.itrhok (mgr.14180) 81 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:08.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:07 smithi084 conmon[33408]: audit 2024-01-13T09:48:07.536870+0000 mon.smithi102 (mon.1) 4 : audit [INF] from='client.? 172.21.15.102:0/295376378' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d524ee64-298b-4417-9902-98f78ab71a6a"}]: dispatch 2024-01-13T09:48:08.278 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:07 smithi084 conmon[33408]: audit 2024-01-13T09:48:07.537312+0000 mon.smithi084 (mon.0) 292 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d524ee64-298b-4417-9902-98f78ab71a6a"}]: dispatch 2024-01-13T09:48:08.278 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:07 smithi084 conmon[33408]: audit 2024-01-13T09:48:07.540670+0000 mon.smithi084 (mon.0) 293 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d524ee64-298b-4417-9902-98f78ab71a6a"}]': finished 2024-01-13T09:48:08.278 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:07 smithi084 conmon[33408]: cluster 2024-01-13T09:48:07.540717+0000 mon.smithi084 (mon.0) 294 : cluster [DBG] osdmap e10: 1 total, 0 up, 1 in 2024-01-13T09:48:08.278 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:07 smithi084 conmon[33408]: audit 2024-01-13T09:48:07.540807+0000 mon.smithi084 (mon.0) 295 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:48:08.278 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:07 smithi084 conmon[33408]: audit 2024-01-13T09:48:07.831029+0000 mon.smithi102 (mon.1) 5 : audit [INF] from='client.? 172.21.15.102:0/231390807' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:48:08.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:07 smithi084 conmon[33408]: 2024-01-13T09:48:08.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:07 smithi084 conmon[33408]: audit 2024-01-13T09:48:07.831487+0000 mon.smithi084 (mon.0) 296 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:48:08.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:07 smithi084 conmon[33408]: audit 2024-01-13T09:48:07.835078+0000 mon.smithi084 (mon.0) 297 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:48:08.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:07 smithi084 conmon[33408]: cluster 2024-01-13T09:48:07.835148+0000 mon.smithi084 (mon.0) 298 : cluster [DBG] osdmap e11: 0 total, 0 up, 0 in 2024-01-13T09:48:08.418 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:48:09.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: cephadm 2024-01-13T09:48:08.488513+0000 mgr.smithi084.itrhok (mgr.14180) 82 : cephadm [ERR] executing create_from_spec_one(([('smithi084', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi102', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:48:09.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:48:09.235 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:48:09.235 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: return f(*arg) 2024-01-13T09:48:09.235 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:48:09.235 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:48:09.235 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:48:09.235 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: code, '\n'.join(err))) 2024-01-13T09:48:09.235 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:48:09.235 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpg8mvdwwu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp8xg7ywzw:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:48:09.235 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:48:09.235 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:48:09.235 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:48:09.235 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new d524ee64-298b-4417-9902-98f78ab71a6a 2024-01-13T09:48:09.235 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:48:09.235 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:48:09.235 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:48:09.236 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:48:09.236 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:48:09.236 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr self.prepare() 2024-01-13T09:48:09.236 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:09.236 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:09.236 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:48:09.236 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:48:09.236 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:48:09.236 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:48:09.236 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:48:09.236 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:48:09.236 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:48:09.236 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:48:09.236 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:48:09.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:48:09.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:48:09.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:48:09.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:48:09.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:48:09.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:48:09.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:48:09.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:48:09.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:48:09.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:48:09.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:48:09.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:48:09.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:48:09.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:48:09.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:48:09.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:48:09.238 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:09.238 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:09.238 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:48:09.238 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:48:09.238 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:48:09.238 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:48:09.238 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:09.238 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:09.238 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:48:09.238 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:48:09.238 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:48:09.238 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:48:09.238 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:48:09.238 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:48:09.238 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:48:09.239 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:48:09.239 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:09.239 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:09.239 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:48:09.239 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:48:09.239 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:48:09.239 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:48:09.239 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:48:09.239 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:48:09.239 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:48:09.239 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: main() 2024-01-13T09:48:09.239 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:48:09.239 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: r = ctx.func(ctx) 2024-01-13T09:48:09.239 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:48:09.240 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:48:09.240 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:48:09.240 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:48:09.240 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:48:09.240 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:48:09.240 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774 2024-01-13T09:48:09.240 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:48:09.240 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:48:09.240 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:48:09.240 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:48:09.240 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:48:09.240 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:48:09.240 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpg8mvdwwu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp8xg7ywzw:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:48:09.240 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: audit 2024-01-13T09:48:08.756534+0000 mon.smithi084 (mon.0) 299 : audit [INF] from='client.? 172.21.15.84:0/3449037531' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ddd82956-f842-4892-bf29-0437deee71f8"}]: dispatch 2024-01-13T09:48:09.240 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: audit 2024-01-13T09:48:08.762182+0000 mon.smithi084 (mon.0) 300 : audit [INF] from='client.? 172.21.15.84:0/3449037531' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ddd82956-f842-4892-bf29-0437deee71f8"}]': finished 2024-01-13T09:48:09.241 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: cluster 2024-01-13T09:48:08.762218+0000 mon.smithi084 (mon.0) 301 : cluster [DBG] osdmap e12: 1 total, 0 up, 1 in 2024-01-13T09:48:09.241 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:08 smithi102 conmon[37544]: audit 2024-01-13T09:48:08.762303+0000 mon.smithi084 (mon.0) 302 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:48:09.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: cephadm 2024-01-13T09:48:08.488513+0000 mgr.smithi084.itrhok (mgr.14180) 82 : cephadm [ERR] executing create_from_spec_one(([('smithi084', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi102', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:48:09.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:48:09.278 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:48:09.278 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: return f(*arg) 2024-01-13T09:48:09.278 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:48:09.278 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:48:09.278 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:48:09.278 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: code, '\n'.join(err))) 2024-01-13T09:48:09.278 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:48:09.278 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpg8mvdwwu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp8xg7ywzw:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:48:09.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:48:09.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:48:09.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:48:09.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new d524ee64-298b-4417-9902-98f78ab71a6a 2024-01-13T09:48:09.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:48:09.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:48:09.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:48:09.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:48:09.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:48:09.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr self.prepare() 2024-01-13T09:48:09.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:09.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:09.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:48:09.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:48:09.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:48:09.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:48:09.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:48:09.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:48:09.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:48:09.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:48:09.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:48:09.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:48:09.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:48:09.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:48:09.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:48:09.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:48:09.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:48:09.281 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:48:09.281 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:48:09.281 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:48:09.281 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:48:09.281 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:48:09.281 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:48:09.281 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:48:09.281 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:48:09.281 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:48:09.281 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:48:09.282 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:09.282 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:09.282 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:48:09.282 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:48:09.282 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:48:09.282 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:48:09.282 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:09.282 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:09.282 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:48:09.283 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:48:09.283 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:48:09.283 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:48:09.283 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:48:09.283 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:48:09.283 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:48:09.283 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:48:09.283 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:09.283 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:09.283 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:48:09.283 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:48:09.283 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:48:09.283 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:48:09.283 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:48:09.284 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:48:09.284 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:48:09.284 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: main() 2024-01-13T09:48:09.284 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:48:09.284 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: r = ctx.func(ctx) 2024-01-13T09:48:09.284 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:48:09.284 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:48:09.284 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:48:09.284 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:48:09.284 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:48:09.284 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:48:09.284 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774 2024-01-13T09:48:09.284 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:48:09.284 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:48:09.285 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:48:09.285 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:48:09.285 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:48:09.285 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:48:09.285 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpg8mvdwwu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp8xg7ywzw:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:48:09.285 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: audit 2024-01-13T09:48:08.756534+0000 mon.smithi084 (mon.0) 299 : audit [INF] from='client.? 172.21.15.84:0/3449037531' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ddd82956-f842-4892-bf29-0437deee71f8"}]: dispatch 2024-01-13T09:48:09.285 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: audit 2024-01-13T09:48:08.762182+0000 mon.smithi084 (mon.0) 300 : audit [INF] from='client.? 172.21.15.84:0/3449037531' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ddd82956-f842-4892-bf29-0437deee71f8"}]': finished 2024-01-13T09:48:09.285 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: cluster 2024-01-13T09:48:08.762218+0000 mon.smithi084 (mon.0) 301 : cluster [DBG] osdmap e12: 1 total, 0 up, 1 in 2024-01-13T09:48:09.285 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:08 smithi084 conmon[33408]: audit 2024-01-13T09:48:08.762303+0000 mon.smithi084 (mon.0) 302 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:48:10.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:09 smithi102 conmon[37544]: cluster 2024-01-13T09:48:08.951550+0000 mgr.smithi084.itrhok (mgr.14180) 83 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:10.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:09 smithi102 conmon[37544]: 2024-01-13T09:48:10.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:09 smithi102 conmon[37544]: audit 2024-01-13T09:48:09.065567+0000 mon.smithi084 (mon.0) 303 : audit [INF] from='client.? 172.21.15.84:0/3858930620' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:48:10.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:09 smithi102 conmon[37544]: audit 2024-01-13T09:48:09.071706+0000 mon.smithi084 (mon.0) 304 : audit 2024-01-13T09:48:10.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:09 smithi102 conmon[37544]: [INF] from='client.? 172.21.15.84:0/3858930620' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:48:10.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:09 smithi102 conmon[37544]: cluster 2024-01-13T09:48:09.071750+0000 mon.smithi084 (mon.0) 305 : cluster [DBG] osdmap e13: 0 total, 0 up, 0 in 2024-01-13T09:48:10.250 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:09 smithi084 conmon[33408]: cluster 2024-01-13T09:48:08.951550+0000 mgr.smithi084.itrhok (mgr.14180) 83 : 2024-01-13T09:48:10.250 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:09 smithi084 conmon[33408]: cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:10.250 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:09 smithi084 conmon[33408]: audit 2024-01-13T09:48:09.065567+0000 mon.smithi084 (mon.0) 303 : audit [INF] from='client.? 172.21.15.84:0/3858930620' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:48:10.250 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:09 smithi084 conmon[33408]: audit 2024-01-13T09:48:09.071706+0000 mon.smithi084 (mon.0) 304 : audit [INF] from='client.? 172.21.15.84:0/3858930620' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:48:10.250 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:09 smithi084 conmon[33408]: cluster 2024-01-13T09:48:09.071750+0000 mon.smithi084 (mon.0) 305 : cluster [DBG] osdmap e13: 0 total, 0 up, 0 in 2024-01-13T09:48:10.868 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:48:11.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: cephadm 2024-01-13T09:48:10.367573+0000 mgr.smithi084.itrhok (mgr.14180) 84 : cephadm [ERR] executing create_from_spec_one(([('smithi084', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi102', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:48:11.485 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:48:11.485 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:48:11.485 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: return f(*arg) 2024-01-13T09:48:11.485 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:48:11.485 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:48:11.485 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:48:11.485 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: code, '\n'.join(err))) 2024-01-13T09:48:11.485 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi084/config 2024-01-13T09:48:11.485 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp2x4fpmu6:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp7oe4a22g:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:48:11.485 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:48:11.485 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:48:11.485 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:48:11.485 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new ddd82956-f842-4892-bf29-0437deee71f8 2024-01-13T09:48:11.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:48:11.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:48:11.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:48:11.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:48:11.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:48:11.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr self.prepare() 2024-01-13T09:48:11.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:11.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:11.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:48:11.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:48:11.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:48:11.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:48:11.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:48:11.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:48:11.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:48:11.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:48:11.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:48:11.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:48:11.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:48:11.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:48:11.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:48:11.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:48:11.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:48:11.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:48:11.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:48:11.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:48:11.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:48:11.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:48:11.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:48:11.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:48:11.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:48:11.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:48:11.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:48:11.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:11.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:11.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:48:11.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:48:11.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:48:11.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:48:11.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:11.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:11.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:48:11.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:48:11.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:48:11.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:48:11.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:48:11.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:48:11.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:48:11.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:48:11.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:11.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:11.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:48:11.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:48:11.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:48:11.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:48:11.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:48:11.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:48:11.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:48:11.489 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: main() 2024-01-13T09:48:11.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:48:11.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: r = ctx.func(ctx) 2024-01-13T09:48:11.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:48:11.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:48:11.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:48:11.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:48:11.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:48:11.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:48:11.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774 2024-01-13T09:48:11.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:48:11.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:48:11.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:48:11.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:48:11.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:48:11.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:48:11.490 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp2x4fpmu6:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp7oe4a22g:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:48:11.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: cephadm 2024-01-13T09:48:10.368766+0000 mgr.smithi084.itrhok (mgr.14180) 85 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T09:48:11.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: service_id: all-available-devices 2024-01-13T09:48:11.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: service_name: osd.all-available-devices 2024-01-13T09:48:11.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: placement: 2024-01-13T09:48:11.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: host_pattern: '*' 2024-01-13T09:48:11.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: spec: 2024-01-13T09:48:11.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: data_devices: 2024-01-13T09:48:11.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: all: true 2024-01-13T09:48:11.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: filter_logic: AND 2024-01-13T09:48:11.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: objectstore: bluestore 2024-01-13T09:48:11.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:48:11.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpg8mvdwwu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp8xg7ywzw:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:48:11.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:48:11.491 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:48:11.492 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:48:11.492 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new d524ee64-298b-4417-9902-98f78ab71a6a 2024-01-13T09:48:11.492 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:48:11.492 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:48:11.492 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:48:11.492 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:48:11.492 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:48:11.492 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr self.prepare() 2024-01-13T09:48:11.492 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:11.492 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:11.492 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:48:11.492 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:48:11.492 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:48:11.492 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:48:11.493 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:48:11.493 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:48:11.493 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:48:11.493 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:48:11.493 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:48:11.493 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:48:11.493 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:48:11.493 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:48:11.493 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:48:11.493 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:48:11.493 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:48:11.493 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:48:11.493 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:48:11.493 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:48:11.493 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:48:11.493 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:48:11.494 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:48:11.494 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:48:11.494 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:48:11.494 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:48:11.494 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:48:11.495 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:11.495 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:11.495 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:48:11.495 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:48:11.495 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:48:11.495 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:48:11.495 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:11.495 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:11.495 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:48:11.495 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:48:11.495 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:48:11.496 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:48:11.496 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:48:11.496 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:48:11.496 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:48:11.496 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:48:11.496 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:11.496 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:11.496 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lv 2024-01-13T09:48:11.496 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: m/zap.py", line 302, in zap_osd 2024-01-13T09:48:11.496 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:48:11.496 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:48:11.496 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:48:11.496 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:48:11.496 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:48:11.497 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:48:11.497 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: main() 2024-01-13T09:48:11.497 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:48:11.497 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: r = ctx.func(ctx) 2024-01-13T09:48:11.497 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:48:11.497 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:48:11.497 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:48:11.497 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:48:11.497 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:48:11.497 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:48:11.497 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:48:11.497 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:48:11.497 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:48:11.497 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:48:11.497 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:48:11.498 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:48:11.498 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpg8mvdwwu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp8xg7ywzw:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:48:11.498 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:48:11.498 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T09:48:11.498 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: if self._apply_service(spec): 2024-01-13T09:48:11.498 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T09:48:11.498 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T09:48:11.498 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T09:48:11.498 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T09:48:11.498 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T09:48:11.498 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T09:48:11.498 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T09:48:11.498 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T09:48:11.498 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T09:48:11.498 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: raise self._value 2024-01-13T09:48:11.499 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T09:48:11.499 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: result = (True, func(*args, **kwds)) 2024-01-13T09:48:11.499 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T09:48:11.499 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: return list(map(*args)) 2024-01-13T09:48:11.499 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:48:11.499 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: return f(*arg) 2024-01-13T09:48:11.499 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:48:11.499 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:48:11.499 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:48:11.499 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: code, '\n'.join(err))) 2024-01-13T09:48:11.499 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:48:11.500 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpg8mvdwwu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp8xg7ywzw:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:48:11.500 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:48:11.500 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:48:11.500 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:48:11.500 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new d524ee64-298b-4417-9902-98f78ab71a6a 2024-01-13T09:48:11.500 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:48:11.500 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:48:11.500 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:48:11.500 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:48:11.500 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:48:11.500 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr self.prepare() 2024-01-13T09:48:11.500 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:11.500 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:11.500 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:48:11.500 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:48:11.501 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:48:11.501 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:48:11.501 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:48:11.501 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:48:11.501 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:48:11.501 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:48:11.501 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:48:11.501 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:48:11.501 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:48:11.501 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:48:11.501 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:48:11.501 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:48:11.501 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:48:11.501 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr F 2024-01-13T09:48:11.501 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: ile "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:48:11.502 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:48:11.502 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:48:11.502 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:48:11.502 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:48:11.502 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:48:11.502 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:48:11.502 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:48:11.502 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:48:11.502 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:48:11.502 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:11.502 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:11.502 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:48:11.502 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:48:11.502 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:48:11.502 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:48:11.502 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:11.503 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:11.503 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:48:11.503 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:48:11.503 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:48:11.503 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:48:11.503 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:48:11.503 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:48:11.503 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:48:11.503 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:48:11.503 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:11.503 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:11.503 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:48:11.503 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:48:11.503 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:48:11.503 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:48:11.504 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:48:11.504 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:48:11.504 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:48:11.504 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: main() 2024-01-13T09:48:11.504 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:48:11.504 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: r = ctx.func(ctx) 2024-01-13T09:48:11.504 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:48:11.504 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:48:11.504 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:48:11.504 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:48:11.504 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:48:11.504 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:48:11.504 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:48:11.504 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:48:11.504 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:48:11.505 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:48:11.505 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:48:11.505 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:48:11.505 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpg8mvdwwu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp8xg7ywzw:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:48:11.505 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: cluster 2024-01-13T09:48:10.369383+0000 mgr.smithi084.itrhok (mgr.14180) 86 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:11.505 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:11 smithi102 conmon[37544]: audit 2024-01-13T09:48:10.878707+0000 mon.smithi084 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.84:0/96769776' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:48:11.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: cephadm 2024-01-13T09:48:10.367573+0000 mgr.smithi084.itrhok (mgr.14180) 84 : cephadm [ERR] executing create_from_spec_one(([('smithi084', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi102', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:48:11.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:48:11.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:48:11.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: return f(*arg) 2024-01-13T09:48:11.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:48:11.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:48:11.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:48:11.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: code, '\n'.join(err))) 2024-01-13T09:48:11.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi084/config 2024-01-13T09:48:11.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp2x4fpmu6:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp7oe4a22g:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:48:11.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:48:11.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:48:11.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:48:11.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new ddd82956-f842-4892-bf29-0437deee71f8 2024-01-13T09:48:11.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:48:11.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:48:11.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:48:11.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:48:11.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:48:11.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr self.prepare() 2024-01-13T09:48:11.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:11.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:11.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:48:11.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:48:11.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:48:11.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:48:11.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:48:11.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:48:11.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:48:11.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:48:11.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:48:11.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:48:11.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:48:11.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:48:11.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:48:11.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:48:11.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:48:11.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:48:11.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:48:11.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:48:11.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:48:11.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:48:11.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:48:11.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:48:11.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:48:11.531 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:48:11.531 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:48:11.531 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:11.531 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:11.531 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:48:11.531 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:48:11.531 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:48:11.531 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:48:11.531 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:11.531 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:11.531 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:48:11.531 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:48:11.531 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:48:11.531 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:48:11.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:48:11.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:48:11.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:48:11.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:48:11.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:11.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:11.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:48:11.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:48:11.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:48:11.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:48:11.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:48:11.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:48:11.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:48:11.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: main() 2024-01-13T09:48:11.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:48:11.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: r = ctx.func(ctx) 2024-01-13T09:48:11.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:48:11.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:48:11.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:48:11.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:48:11.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:48:11.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:48:11.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774 2024-01-13T09:48:11.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:48:11.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:48:11.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:48:11.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:48:11.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:48:11.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:48:11.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp2x4fpmu6:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp7oe4a22g:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:48:11.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: cephadm 2024-01-13T09:48:10.368766+0000 mgr.smithi084.itrhok (mgr.14180) 85 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T09:48:11.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: service_id: all-available-devices 2024-01-13T09:48:11.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: service_name: osd.all-available-devices 2024-01-13T09:48:11.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: placement: 2024-01-13T09:48:11.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: host_pattern: '*' 2024-01-13T09:48:11.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: spec: 2024-01-13T09:48:11.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: data_devices: 2024-01-13T09:48:11.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: all: true 2024-01-13T09:48:11.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: filter_logic: AND 2024-01-13T09:48:11.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: objectstore: bluestore 2024-01-13T09:48:11.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:48:11.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpg8mvdwwu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp8xg7ywzw:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:48:11.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:48:11.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:48:11.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:48:11.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new d524ee64-298b-4417-9902-98f78ab71a6a 2024-01-13T09:48:11.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:48:11.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:48:11.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:48:11.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:48:11.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:48:11.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr self.prepare() 2024-01-13T09:48:11.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:11.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:11.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:48:11.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:48:11.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:48:11.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:48:11.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:48:11.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:48:11.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:48:11.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:48:11.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:48:11.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:48:11.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:48:11.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:48:11.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:48:11.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:48:11.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:48:11.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:48:11.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:48:11.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:48:11.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:48:11.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:48:11.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:48:11.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:48:11.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:48:11.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:48:11.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:48:11.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:11.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:11.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:48:11.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:48:11.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:48:11.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:48:11.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:11.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:11.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:48:11.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:48:11.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:48:11.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:48:11.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:48:11.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:48:11.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:48:11.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:48:11.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:11.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:11.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lv 2024-01-13T09:48:11.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: m/zap.py", line 302, in zap_osd 2024-01-13T09:48:11.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:48:11.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:48:11.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:48:11.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:48:11.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:48:11.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:48:11.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: main() 2024-01-13T09:48:11.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:48:11.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: r = ctx.func(ctx) 2024-01-13T09:48:11.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:48:11.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:48:11.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:48:11.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:48:11.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:48:11.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:48:11.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:48:11.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:48:11.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:48:11.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:48:11.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:48:11.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:48:11.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpg8mvdwwu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp8xg7ywzw:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:48:11.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:48:11.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T09:48:11.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: if self._apply_service(spec): 2024-01-13T09:48:11.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T09:48:11.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T09:48:11.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T09:48:11.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T09:48:11.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T09:48:11.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T09:48:11.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T09:48:11.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T09:48:11.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T09:48:11.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: raise self._value 2024-01-13T09:48:11.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T09:48:11.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: result = (True, func(*args, **kwds)) 2024-01-13T09:48:11.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T09:48:11.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: return list(map(*args)) 2024-01-13T09:48:11.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:48:11.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: return f(*arg) 2024-01-13T09:48:11.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:48:11.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:48:11.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:48:11.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: code, '\n'.join(err))) 2024-01-13T09:48:11.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:48:11.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpg8mvdwwu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp8xg7ywzw:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:48:11.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:48:11.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:48:11.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:48:11.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new d524ee64-298b-4417-9902-98f78ab71a6a 2024-01-13T09:48:11.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:48:11.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:48:11.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:48:11.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:48:11.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:48:11.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr self.prepare() 2024-01-13T09:48:11.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:11.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:11.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:48:11.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:48:11.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:48:11.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:48:11.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:48:11.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:48:11.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:48:11.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:48:11.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:48:11.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:48:11.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:48:11.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:48:11.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:48:11.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:48:11.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:48:11.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr F 2024-01-13T09:48:11.545 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: ile "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:48:11.545 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:48:11.545 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:48:11.545 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:48:11.545 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:48:11.545 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:48:11.545 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:48:11.545 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:48:11.545 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:48:11.545 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:48:11.545 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:11.545 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:11.545 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:48:11.545 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:48:11.545 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:48:11.545 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:48:11.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:11.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:11.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:48:11.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:48:11.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:48:11.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:48:11.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:48:11.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:48:11.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:48:11.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:48:11.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:48:11.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:48:11.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:48:11.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:48:11.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:48:11.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:48:11.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:48:11.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:48:11.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:48:11.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: main() 2024-01-13T09:48:11.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:48:11.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: r = ctx.func(ctx) 2024-01-13T09:48:11.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:48:11.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:48:11.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:48:11.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:48:11.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:48:11.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:48:11.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:48:11.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:48:11.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:48:11.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:48:11.548 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:48:11.548 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:48:11.548 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpg8mvdwwu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp8xg7ywzw:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:48:11.548 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: cluster 2024-01-13T09:48:10.369383+0000 mgr.smithi084.itrhok (mgr.14180) 86 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:11.548 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:11 smithi084 conmon[33408]: audit 2024-01-13T09:48:10.878707+0000 mon.smithi084 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.84:0/96769776' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:48:11.604 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:48:12.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:12 smithi102 conmon[37544]: cluster 2024-01-13T09:48:11.232340+0000 mon.smithi084 (mon.0) 307 : 2024-01-13T09:48:12.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:12 smithi102 conmon[37544]: cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T09:48:12.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:12 smithi102 conmon[37544]: audit 2024-01-13T09:48:12.205937+0000 mon.smithi084 (mon.0) 308 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:48:12.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:12 smithi084 conmon[33408]: cluster 2024-01-13T09:48:11.232340+0000 mon.smithi084 (mon.0) 307 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T09:48:12.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:12 smithi084 conmon[33408]: audit 2024-01-13T09:48:12.205937+0000 mon.smithi084 (mon. 2024-01-13T09:48:12.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:12 smithi084 conmon[33408]: 0) 308 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:48:12.604 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:48:13.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:13 smithi084 conmon[33408]: cluster 2024-01-13T09:48:12.369858+0000 mgr.smithi084.itrhok (mgr.14180) 87 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:13.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:13 smithi102 conmon[37544]: cluster 2024-01-13T09:48:12.369858+0000 mgr.smithi084.itrhok (mgr 2024-01-13T09:48:13.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:13 smithi102 conmon[37544]: .14180) 87 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:14.749 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:48:15.470 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:48:15.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:15 smithi102 conmon[37544]: cluster 2024-01-13T09:48:14.370445+0000 mgr.smithi084.itrhok (mgr.14180) 88 2024-01-13T09:48:15.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:15 smithi102 conmon[37544]: : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:15.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:15 smithi102 conmon[37544]: audit 2024-01-13T09:48:14.759729+0000 mon.smithi084 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.84:0/464462698' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:48:15.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:15 smithi084 conmon[33408]: cluster 2024-01-13T09:48:14.370445+0000 mgr.smithi084.itrhok (mgr.14180) 88 : cluster 2024-01-13T09:48:15.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:15 smithi084 conmon[33408]: [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:15.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:15 smithi084 conmon[33408]: audit 2024-01-13T09:48:14.759729+0000 mon.smithi084 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.84:0/464462698' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:48:16.471 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:48:17.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:17 smithi102 conmon[37544]: cluster 2024-01-13T09:48:16.370990+0000 mgr.smithi084.itrhok (mgr.14180) 89 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:17.776 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:17 smithi084 conmon[33408]: cluster 2024-01-13T09:48:16.370990+0000 mgr.smithi084.itrhok (mgr.14180) 89 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:18.455 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:48:19.017 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:48:19.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:19 smithi102 conmon[37544]: cluster 2024-01-13T09:48:18.371396+0000 mgr.smithi084.itrhok (mgr.14180) 90 2024-01-13T09:48:19.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:19 smithi102 conmon[37544]: : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:19.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:19 smithi102 conmon[37544]: audit 2024-01-13T09:48:18.465512+0000 mon.smithi084 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.84:0/4255797717' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:48:19.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:19 smithi084 conmon[33408]: cluster 2024-01-13T09:48:18.371396+0000 mgr.smithi084.itrhok (mgr.14180) 90 : cluster 2024-01-13T09:48:19.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:19 smithi084 conmon[33408]: [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:19.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:19 smithi084 conmon[33408]: audit 2024-01-13T09:48:18.465512+0000 mon.smithi084 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.84:0/4255797717' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:48:20.019 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:48:21.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:21 smithi102 conmon[37544]: cluster 2024-01-13T09:48:20.371833+0000 mgr.smithi084.itrhok (mgr.14180) 91 : cluster 2024-01-13T09:48:21.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:21 smithi102 conmon[37544]: [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:21.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:21 smithi084 conmon[33408]: cluster 2024-01-13T09:48:20.371833+0000 mgr.smithi084.itrhok (mgr.14180) 91 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:21.984 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:48:22.276 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:22 smithi084 conmon[33408]: audit 2024-01-13T09:48:21.994964+0000 mon.smithi084 (mon.0) 311 2024-01-13T09:48:22.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:22 smithi084 conmon[33408]: : audit [DBG] from='client.? 172.21.15.84:0/2556636830' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:48:22.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:22 smithi102 conmon[37544]: audit 2024-01-13T09:48:21.994964+0000 mon.smithi084 (mon.0) 311 : audit 2024-01-13T09:48:22.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:22 smithi102 conmon[37544]: [DBG] from='client.? 172.21.15.84:0/2556636830' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:48:22.682 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:48:23.526 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:23 smithi084 conmon[33408]: cluster 2024-01-13T09:48:22.372511+0000 mgr.smithi084.itrhok (mgr.14180) 92 : cluster 2024-01-13T09:48:23.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:23 smithi084 conmon[33408]: [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:23.684 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:48:23.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:23 smithi102 conmon[37544]: cluster 2024-01-13T09:48:22.372511+0000 mgr.smithi084.itrhok (mgr.14180) 92 2024-01-13T09:48:23.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:23 smithi102 conmon[37544]: : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:25.627 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:48:25.647 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:25 smithi084 conmon[33408]: cluster 2024-01-13T09:48:24.372864+0000 mgr.smithi084.itrhok (mgr.14180) 93 : 2024-01-13T09:48:25.647 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:25 smithi084 conmon[33408]: cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:25.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:25 smithi102 conmon[37544]: cluster 2024-01-13T09:48:24.372864+0000 mgr.smithi084.itrhok (mgr.14180) 93 : cluster 2024-01-13T09:48:25.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:25 smithi102 conmon[37544]: [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:26.255 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:48:26.526 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:26 smithi084 conmon[33408]: audit 2024-01-13T09:48:25.637138+0000 mon.smithi084 (mon.0) 312 : audit [DBG] 2024-01-13T09:48:26.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:26 smithi084 conmon[33408]: from='client.? 172.21.15.84:0/1578462935' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:48:26.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:26 smithi102 conmon[37544]: audit 2024-01-13T09:48:25.637138+0000 mon.smithi084 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.84:0/1578462935' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:48:26.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:26 smithi102 conmon[37544]: 2024-01-13T09:48:27.257 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:48:27.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:27 smithi102 conmon[37544]: cluster 2024-01-13T09:48:26.373240+0000 mgr.smithi084.itrhok (mgr.14180) 94 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:27.740 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:27 smithi084 conmon[33408]: cluster 2024-01-13T09:48:26.373240+0000 mgr.smithi084.itrhok 2024-01-13T09:48:27.740 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:27 smithi084 conmon[33408]: (mgr.14180) 94 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:29.143 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:48:29.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:29 smithi102 conmon[37544]: cluster 2024-01-13T09:48:28.373830+0000 mgr.smithi084.itrhok (mgr.14180) 95 : cluster [DBG] 2024-01-13T09:48:29.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:29 smithi102 conmon[37544]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:29.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:29 smithi102 conmon[37544]: audit 2024-01-13T09:48:29.152459+0000 mon.smithi084 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.84:0/2875582139' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:48:29.776 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:29 smithi084 conmon[33408]: cluster 2024-01-13T09:48:28.373830+0000 mgr.smithi084.itrhok (mgr 2024-01-13T09:48:29.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:29 smithi084 conmon[33408]: .14180) 95 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:29.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:29 smithi084 conmon[33408]: audit 2024-01-13T09:48:29.152459+0000 mon.smithi084 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.84:0/2875582139' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:48:29.908 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:48:30.909 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:48:31.483 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:31 smithi102 conmon[37544]: cluster 2024-01-13T09:48:30.374194+0000 mgr.smithi084.itrhok (mgr.14180) 96 : 2024-01-13T09:48:31.484 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:31 smithi102 conmon[37544]: cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:31.526 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:31 smithi084 conmon[33408]: cluster 2024-01-13T09:48:30.374194+0000 mgr.smithi084.itrhok (mgr.14180) 96 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:32.744 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:48:33.388 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:48:33.733 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:33 smithi102 conmon[37544]: cluster 2024-01-13T09:48:32.374808+0000 mgr.smithi084.itrhok (mgr.14180) 97 : cluster 2024-01-13T09:48:33.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:33 smithi102 conmon[37544]: [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:33.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:33 smithi102 conmon[37544]: audit 2024-01-13T09:48:32.754294+0000 mon.smithi084 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.84:0/3491013554' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:48:33.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:33 smithi084 conmon[33408]: cluster 2024-01-13T09:48:32.374808+0000 mgr.smithi084.itrhok (mgr.14180) 97 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:33.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:33 smithi084 conmon[33408]: audit 2024-01-13T09:48:32.754294+0000 mon.smithi084 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.84:0/3491013554' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:48:34.390 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:48:35.733 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:35 smithi102 conmon[37544]: cluster 2024-01-13T09:48:34.375250+0000 mgr.smithi084.itrhok (mgr.14180) 2024-01-13T09:48:35.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:35 smithi102 conmon[37544]: 98 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:35.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:35 smithi084 conmon[33408]: cluster 2024-01-13T09:48:34.375250+0000 mgr.smithi084.itrhok (mgr.14180) 98 : 2024-01-13T09:48:35.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:35 smithi084 conmon[33408]: cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:36.416 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:48:36.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:36 smithi102 conmon[37544]: audit 2024-01-13T09:48:36.426807+0000 mon.smithi084 (mon.0) 2024-01-13T09:48:36.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:36 smithi102 conmon[37544]: 315 : audit [DBG] from='client.? 172.21.15.84:0/3302844698' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:48:36.776 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:36 smithi084 conmon[33408]: audit 2024-01-13T09:48:36.426807+0000 mon.smithi084 (mon 2024-01-13T09:48:36.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:36 smithi084 conmon[33408]: .0) 315 : audit [DBG] from='client.? 172.21.15.84:0/3302844698' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:48:37.013 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:48:37.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:37 smithi102 conmon[37544]: cluster 2024-01-13T09:48:36.375645+0000 mgr.smithi084.itrhok (mgr.14180) 99 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:37.776 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:37 smithi084 conmon[33408]: cluster 2024-01-13T09:48:36.375645+0000 mgr.smithi084.itrhok (mgr.14180) 99 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:38.014 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:48:39.564 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:39 smithi084 conmon[33408]: cluster 2024-01-13T09:48:38.376327+0000 mgr.smithi084.itrhok (mgr.14180) 2024-01-13T09:48:39.564 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:39 smithi084 conmon[33408]: 100 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:39.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:39 smithi102 conmon[37544]: cluster 2024-01-13T09:48:38.376327+0000 mgr.smithi084.itrhok (mgr.14180) 100 : cluster 2024-01-13T09:48:39.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:39 smithi102 conmon[37544]: [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:39.918 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:48:40.512 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:48:40.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:40 smithi102 conmon[37544]: audit 2024-01-13T09:48:39.928316+0000 mon.smithi084 (mon.0) 316 : audit [DBG] 2024-01-13T09:48:40.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:40 smithi102 conmon[37544]: from='client.? 172.21.15.84:0/480157641' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:48:40.776 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:40 smithi084 conmon[33408]: audit 2024-01-13T09:48:39.928316+0000 mon.smithi084 (mon.0) 316 2024-01-13T09:48:40.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:40 smithi084 conmon[33408]: : audit [DBG] from='client.? 172.21.15.84:0/480157641' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:48:41.514 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:48:41.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:41 smithi102 conmon[37544]: cluster 2024-01-13T09:48:40.376755+0000 mgr.smithi084.itrhok (mgr.14180) 101 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:41.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:41 smithi084 conmon[33408]: cluster 2024-01-13T09:48:40.376755+0000 mgr.smithi084.itrhok (mgr.14180) 101 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:43.309 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:48:43.733 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:43 smithi102 conmon[37544]: cluster 2024-01-13T09:48:42.377118+0000 mgr.smithi084.itrhok (mgr.14180) 102 : cluster 2024-01-13T09:48:43.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:43 smithi102 conmon[37544]: [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:43.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:43 smithi102 conmon[37544]: audit 2024-01-13T09:48:43.319711+0000 mon.smithi084 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.84:0/383956005' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:48:43.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:43 smithi084 conmon[33408]: cluster 2024-01-13T09:48:42.377118+0000 mgr.smithi084.itrhok (mgr.14180) 102 : cluster [DBG] 2024-01-13T09:48:43.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:43 smithi084 conmon[33408]: pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:43.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:43 smithi084 conmon[33408]: audit 2024-01-13T09:48:43.319711+0000 mon.smithi084 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.84:0/383956005' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:48:44.019 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:48:45.020 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:48:45.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:45 smithi102 conmon[37544]: cluster 2024-01-13T09:48:44.377611+0000 mgr.smithi084.itrhok (mgr.14180) 103 : cluster [DBG] 2024-01-13T09:48:45.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:45 smithi102 conmon[37544]: pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:45.776 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:45 smithi084 conmon[33408]: cluster 2024-01-13T09:48:44.377611+0000 mgr.smithi084.itrhok (mgr.14180) 103 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:46.932 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:48:47.510 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:48:47.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:47 smithi102 conmon[37544]: cluster 2024-01-13T09:48:46.378192+0000 mgr.smithi084.itrhok (mgr.14180) 104 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:47.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:47 smithi102 conmon[37544]: audit 2024-01-13T09:48:46.942293+0000 mon.smithi084 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.84:0/2067839538' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:48:47.776 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:47 smithi084 conmon[33408]: cluster 2024-01-13T09:48:46.378192+0000 mgr.smithi084.itrhok (mgr 2024-01-13T09:48:47.776 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:47 smithi084 conmon[33408]: .14180) 104 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:47.776 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:47 smithi084 conmon[33408]: audit 2024-01-13T09:48:46.942293+0000 mon.smithi084 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.84:0/2067839538' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:48:48.511 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:48:49.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:49 smithi102 conmon[37544]: cluster 2024-01-13T09:48:48.378682+0000 mgr.smithi084.itrhok (mgr.14180) 105 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:49.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:49 smithi084 conmon[33408]: cluster 2024-01-13T09:48:48.378682+0000 mgr.smithi084.itrhok (mgr.14180) 105 : 2024-01-13T09:48:49.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:49 smithi084 conmon[33408]: cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:50.510 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:48:51.084 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:48:51.526 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:51 smithi084 conmon[33408]: cluster 2024-01-13T09:48:50.379079+0000 mgr.smithi084.itrhok (mgr.14180) 106 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:51.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:51 smithi084 conmon[33408]: audit 2024-01-13T09:48:50.520318+0000 mon.smithi084 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.84:0/231971071' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:48:51.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:51 smithi102 conmon[37544]: cluster 2024-01-13T09:48:50.379079+0000 mgr.smithi084.itrhok (mgr.14180) 106 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:51.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:51 smithi102 conmon[37544]: audit 2024-01-13T09:48:50.520318+0000 mon.smithi084 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.84:0/231971071' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:48:52.086 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:48:53.574 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:53 smithi084 conmon[33408]: cluster 2024-01-13T09:48:52.379490+0000 mgr.smithi084.itrhok (mgr.14180) 107 : cluster 2024-01-13T09:48:53.574 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:53 smithi084 conmon[33408]: [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:53.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:53 smithi102 conmon[37544]: cluster 2024-01-13T09:48:52.379490+0000 mgr.smithi084.itrhok (mgr.14180) 107 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:53.899 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:48:54.601 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:48:54.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:54 smithi102 conmon[37544]: audit 2024-01-13T09:48:53.909694+0000 mon.smithi084 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.84:0/2518078259' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:48:54.776 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:54 smithi084 conmon[33408]: audit 2024-01-13T09:48:53.909694+0000 mon.smithi084 (mon. 2024-01-13T09:48:54.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:54 smithi084 conmon[33408]: 0) 320 : audit [DBG] from='client.? 172.21.15.84:0/2518078259' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:48:55.602 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:48:55.733 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:55 smithi102 conmon[37544]: cluster 2024-01-13T09:48:54.379902+0000 mgr.smithi084.itrhok (mgr.14180) 108 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:55.776 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:55 smithi084 conmon[33408]: cluster 2024-01-13T09:48:54.379902+0000 mgr.smithi084.itrhok (mgr.14180) 108 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:57.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:57 smithi084 conmon[33408]: cluster 2024-01-13T09:48:56. 2024-01-13T09:48:57.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:57 smithi084 conmon[33408]: 380311+0000 mgr.smithi084.itrhok (mgr.14180) 109 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:57.564 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:48:57.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:57 smithi102 conmon[37544]: cluster 2024-01-13T09:48:56.380311+0000 mgr.smithi084.itrhok (mgr.14180) 109 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:58.274 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:48:58.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:58 smithi084 conmon[33408]: audit 2024-01-13T09:48:57.574442+0000 mon.smithi084 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.84:0/373793663' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:48:58.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:58 smithi102 conmon[37544]: audit 2024-01-13T09:48:57.574442+0000 mon.smithi084 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.84:0/373793663' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:48:59.276 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:48:59.725 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:59 smithi084 conmon[33408]: cluster 2024-01-13T09:48:58.380793+0000 mgr.smithi084.itrhok (mgr.14180) 110 : 2024-01-13T09:48:59.725 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:48:59 smithi084 conmon[33408]: cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:48:59.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:48:59 smithi102 conmon[37544]: cluster 2024-01-13T09:48:58.380793+0000 mgr.smithi084.itrhok (mgr.14180) 110 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:01.141 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:49:01.526 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:01 smithi084 conmon[33408]: cluster 2024-01-13T09:49:00.381423+0000 mgr.smithi084.itrhok (mgr.14180) 111 2024-01-13T09:49:01.526 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:01 smithi084 conmon[33408]: : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:01.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:01 smithi084 conmon[33408]: audit 2024-01-13T09:49:01.151802+0000 mon.smithi084 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.84:0/422080399' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:49:01.733 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:01 smithi102 conmon[37544]: cluster 2024-01-13T09:49:00.381423+0000 mgr.smithi084.itrhok (mgr.14180) 111 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:01.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:01 smithi102 conmon[37544]: audit 2024-01-13T09:49:01.151802+0000 mon.smithi084 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.84:0/422080399' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:49:01.807 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:49:02.808 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:49:03.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:03 smithi102 conmon[37544]: cluster 2024-01-13T09:49:02.381966+0000 mgr.smithi084.itrhok (mgr.14180) 112 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:03.760 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:03 smithi084 conmon[33408]: cluster 2024-01-13T09:49:02.381966+0000 mgr.smithi084.itrhok (mgr.14180) 112 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:04.770 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:49:05.405 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:49:05.733 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:05 smithi102 conmon[37544]: cluster 2024-01-13T09:49:04.382374+0000 mgr.smithi084.itrhok (mgr.14180) 113 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:05.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:05 smithi102 conmon[37544]: audit 2024-01-13T09:49:04.781062 2024-01-13T09:49:05.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:05 smithi102 conmon[37544]: +0000 mon.smithi084 (mon.0) 323 : audit [DBG] from='client.? 172.21.15.84:0/794686713' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:49:05.776 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:05 smithi084 conmon[33408]: cluster 2024-01-13T09:49:04.382374+0000 mgr.smithi084.itrhok (mgr.14180) 113 : cluster 2024-01-13T09:49:05.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:05 smithi084 conmon[33408]: [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:05.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:05 smithi084 conmon[33408]: audit 2024-01-13T09:49:04.781062+0000 mon.smithi084 (mon.0) 323 : audit [DBG] from='client.? 172.21.15.84:0/794686713' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:49:06.406 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:49:07.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:07 smithi084 conmon[33408]: cluster 2024-01-13T09:49:06.382824+0000 mgr.smithi084.itrhok (mgr.14180) 114 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:07.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:07 smithi102 conmon[37544]: cluster 2024-01-13T09:49:06.382824+0000 mgr.smithi084.itrhok (mgr.14180) 114 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:08.322 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:49:08.733 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:08 smithi102 conmon[37544]: audit 2024-01-13T09:49:08.332268+0000 mon.smithi084 (mon.0) 324 : audit [DBG] from='client.? 172.21.15.84:0/2334919900' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:49:08.776 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:08 smithi084 conmon[33408]: audit 2024-01-13T09:49:08.332268+0000 mon.smithi084 2024-01-13T09:49:08.776 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:08 smithi084 conmon[33408]: (mon.0) 324 : audit [DBG] from='client.? 172.21.15.84:0/2334919900' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:49:08.955 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:49:09.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:09 smithi102 conmon[37544]: cluster 2024-01-13T09:49:08.383371+0000 mgr.smithi084.itrhok ( 2024-01-13T09:49:09.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:09 smithi102 conmon[37544]: mgr.14180) 115 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:09.776 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:09 smithi084 conmon[33408]: cluster 2024-01-13T09:49:08.383371+0000 mgr.smithi084.itrhok (mgr.14180) 115 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:09.956 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:49:11.394 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:11 smithi084 conmon[33408]: audit 2024-01-13T09:49:10.370303+0000 mon.smithi084 (mon.0) 325 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T09:49:11.394 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:11 smithi084 conmon[33408]: cluster 2024-01-13T09:49:10.383723+0000 mgr.smithi084.itrhok (mgr.14180) 116 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:11.733 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:11 smithi102 conmon[37544]: audit 2024-01-13T09:49:10.370303+0000 mon.smithi084 (mon.0) 325 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T09:49:11.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:11 smithi102 conmon[37544]: cluster 2024-01-13T09:49:10.383723+0000 mgr.smithi084.itrhok (mgr.14180) 116 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:12.095 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:49:12.779 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:49:13.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:12 smithi084 conmon[33408]: audit 2024-01-13T09:49:11.746314+0000 mon.smithi084 (mon.0) 326 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:49:13.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:12 smithi084 conmon[33408]: audit 2024-01-13T09:49:12.105540+0000 mon.smithi084 (mon.0) 327 : audit [DBG] from='client.? 172.21.15.84:0/3096830430' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:49:13.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:12 smithi084 conmon[33408]: cluster 2024-01-13T09:49:12.384217+0000 mgr.smithi084.itrhok (mgr.14180) 117 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:13.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:12 smithi102 conmon[37544]: audit 2024-01-13T09:49:11.746314+0000 mon.smithi084 (mon.0) 326 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:49:13.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:12 smithi102 conmon[37544]: audit 2024-01-13T09:49:12.105540+0000 mon.smithi084 (mon.0) 327 : audit [DBG] from='client.? 172.21.15.84:0/3096830430' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:49:13.234 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:12 smithi102 conmon[37544]: cluster 2024-01-13T09:49:12.384217+0000 mgr.smithi084.itrhok (mgr.14180) 117 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:13.780 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:49:15.680 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:15 smithi084 conmon[33408]: cluster 2024-01-13T09:49:14.384715+0000 mgr.smithi084.itrhok (mgr.14180) 118 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:15.680 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:15 smithi084 conmon[33408]: audit 2024-01-13T09:49:15.210156+0000 mon.smithi084 (mon.0) 328 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:49:15.733 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:15 smithi102 conmon[37544]: cluster 2024-01-13T09:49:14.384715+0000 mgr.smithi084.itrhok (mgr.14180) 118 2024-01-13T09:49:15.733 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:15 smithi102 conmon[37544]: : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:15.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:15 smithi102 conmon[37544]: audit 2024-01-13T09:49:15.210156+0000 mon.smithi084 (mon.0) 328 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:49:16.057 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:49:16.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:16 smithi102 conmon[37544]: audit 2024-01-13T09:49:15.558353+0000 mon.smithi084 (mon.0) 329 : audit 2024-01-13T09:49:16.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:16 smithi102 conmon[37544]: [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:49:16.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:16 smithi102 conmon[37544]: audit 2024-01-13T09:49:15.558988+0000 mon.smithi084 (mon.0) 330 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T09:49:16.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:16 smithi102 conmon[37544]: audit 2024-01-13T09:49:15.565076+0000 mon.smithi084 (mon.0) 331 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:49:16.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:16 smithi102 conmon[37544]: cluster 2024-01-13T09:49:15.565366+0000 mgr.smithi084.itrhok (mgr.14180) 119 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:16.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:16 smithi102 conmon[37544]: audit 2024-01-13T09:49:15.568911+0000 mon.smithi084 (mon.0) 332 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T09:49:16.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:16 smithi102 conmon[37544]: audit 2024- 2024-01-13T09:49:16.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:16 smithi102 conmon[37544]: 01-13T09:49:15.571985+0000 mon.smithi084 (mon.0) 333 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:49:16.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:16 smithi102 conmon[37544]: audit 2024-01-13T09: 2024-01-13T09:49:16.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:16 smithi102 conmon[37544]: 49:15.572759+0000 mon.smithi084 (mon 2024-01-13T09:49:16.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:16 smithi102 conmon[37544]: .0) 334 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:49:16.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:16 smithi102 conmon[37544]: audit 2024-01-13T09:49:16.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:16 smithi102 conmon[37544]: 2024-01- 2024-01-13T09:49:16.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:16 smithi102 conmon[37544]: 13T09:49:15.572949+0000 2024-01-13T09:49:16.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:16 smithi102 conmon[37544]: mon.smithi084 (mon.0) 335 : audit 2024-01-13T09:49:16.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:16 smithi102 conmon[37544]: [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:49:16.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:16 smithi102 conmon[37544]: audit 2024- 2024-01-13T09:49:16.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:16 smithi102 conmon[37544]: 01-13T09:49:15.573384 2024-01-13T09:49:16.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:16 smithi102 conmon[37544]: +0000 mon.smithi084 (mon.0) 336 2024-01-13T09:49:16.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:16 smithi102 conmon[37544]: : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:49:16.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:16 smithi102 conmon[37544]: audit 2024-01-13T09:49:16.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:16 smithi102 conmon[37544]: 2024-01-13T09:49 2024-01-13T09:49:16.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:16 smithi102 conmon[37544]: :16.067530+0000 mon.smithi084 (mon 2024-01-13T09:49:16.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:16 smithi102 conmon[37544]: .0) 337 : audit [DBG] from='client.? 172.21.15.84:0/2188995491' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:49:16.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:16 smithi102 conmon[37544]: 2024-01-13T09:49:16.776 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:16 smithi084 conmon[33408]: audit 2024-01-13T09:49:15.558353+0000 mon.smithi084 (mon.0) 329 2024-01-13T09:49:16.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:16 smithi084 conmon[33408]: : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:49:16.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:16 smithi084 conmon[33408]: audit 2024-01-13T09:49:15.558988+0000 mon.smithi084 (mon.0) 330 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T09:49:16.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:16 smithi084 conmon[33408]: audit 2024-01-13T09:49:15.565076+0000 mon.smithi084 (mon.0) 331 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:49:16.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:16 smithi084 conmon[33408]: cluster 2024-01-13T09:49:15.565366+0000 mgr.smithi084.itrhok (mgr.14180) 119 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:16.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:16 smithi084 conmon[33408]: audit 2024-01-13T09:49:15.568911+0000 mon.smithi084 (mon.0) 332 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T09:49:16.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:16 smithi084 conmon[33408]: audit 2024-01-13T09:49:15.571985+0000 mon.smithi084 (mon.0) 333 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:49:16.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:16 smithi084 conmon[33408]: audit 2024-01-13T09:49:15.572759+0000 mon.smithi084 (mon.0) 334 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:49:16.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:16 smithi084 conmon[33408]: audit 2024-01-13T09:49:15.572949+0000 mon.smithi084 (mon.0) 335 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:49:16.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:16 smithi084 conmon[33408]: audit 2024-01-13T09:49:15.573384+0000 mon.smithi084 (mon.0) 336 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:49:16.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:16 smithi084 conmon[33408]: audit 2024-01-13T09:49:16.067530+0000 mon.smithi084 (mon.0) 337 : audit [DBG] from='client.? 172.21.15.84:0/2188995491' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:49:16.852 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:49:17.647 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:17 smithi084 conmon[33408]: cluster 2024-01-13T09:49:16.563296+0000 mon.smithi084 (mon. 2024-01-13T09:49:17.647 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:17 smithi084 conmon[33408]: 0) 338 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T09:49:17.647 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:17 smithi084 conmon[33408]: cluster 2024-01-13T09:49:16.563328+0000 mon.smithi084 (mon.0) 339 : cluster [INF] Cluster is now healthy 2024-01-13T09:49:17.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:17 smithi102 conmon[37544]: cluster 2024-01-13T09:49:16.563296+0000 mon.smithi084 (mon.0) 338 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T09:49:17.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:17 smithi102 conmon[37544]: cluster 2024-01-13T09:49:16.563328+0000 mon.smithi084 (mon.0) 339 : cluster [INF] Cluster is now healthy 2024-01-13T09:49:17.853 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:49:18.705 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:18 smithi084 conmon[33408]: cluster 2024-01-13T09:49:17.565583+0000 mgr.smithi084.itrhok (mgr.14180) 120 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:18.705 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:18 smithi084 conmon[33408]: audit 2024-01-13T09:49:18.173997+0000 mon.smithi102 (mon.1) 6 : audit [INF] from='client.? 172.21.15.102:0/2047219862' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a5d686c-e081-4e54-90c6-9f645c8b5710"}]: dispatch 2024-01-13T09:49:18.705 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:18 smithi084 conmon[33408]: audit 2024-01-13T09:49:18.174533+0000 mon.smithi084 (mon.0) 340 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a5d686c-e081-4e54-90c6-9f645c8b5710"}]: dispatch 2024-01-13T09:49:18.705 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:18 smithi084 conmon[33408]: audit 2024-01-13T09:49:18.178243+0000 mon.smithi084 (mon.0) 341 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1a5d686c-e081-4e54-90c6-9f645c8b5710"}]': finished 2024-01-13T09:49:18.706 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:18 smithi084 conmon[33408]: cluster 2024-01-13T09:49:18.178295+0000 mon.smithi084 (mon.0) 342 : cluster [DBG] osdmap e14: 1 total, 0 up, 1 in 2024-01-13T09:49:18.706 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:18 smithi084 conmon[33408]: audit 2024-01-13T09:49:18.178400+0000 mon.smithi084 (mon.0) 343 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:49:18.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:18 smithi102 conmon[37544]: cluster 2024-01-13T09:49:17.565583+0000 mgr.smithi084.itrhok (mgr.14180) 120 : 2024-01-13T09:49:18.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:18 smithi102 conmon[37544]: cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:18.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:18 smithi102 conmon[37544]: audit 2024-01-13T09:49:18.173997+0000 mon.smithi102 (mon.1) 6 : audit [INF] from='client.? 172.21.15.102:0/2047219862' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a5d686c-e081-4e54-90c6-9f645c8b5710"}]: dispatch 2024-01-13T09:49:18.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:18 smithi102 conmon[37544]: audit 2024-01-13T09:49:18.174533+0000 mon.smithi084 (mon.0) 340 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a5d686c-e081-4e54-90c6-9f645c8b5710"}]: dispatch 2024-01-13T09:49:18.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:18 smithi102 conmon[37544]: audit 2024-01-13T09:49:18.178243+0000 mon.smithi084 (mon.0) 341 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1a5d686c-e081-4e54-90c6-9f645c8b5710"}]': finished 2024-01-13T09:49:18.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:18 smithi102 conmon[37544]: cluster 2024-01-13T09:49:18.178295+0000 mon.smithi084 (mon.0) 342 : cluster [DBG] osdmap e14: 1 total, 0 up, 1 in 2024-01-13T09:49:18.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:18 smithi102 conmon[37544]: audit 2024-01-13T09:49:18.178400+0000 mon.smithi084 (mon.0) 343 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:49:19.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:19 smithi102 conmon[37544]: audit 2024-01-13T09:49:18.465028+0000 mon.smithi102 (mon.1) 7 : audit 2024-01-13T09:49:19.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:19 smithi102 conmon[37544]: [INF] from='client.? 172.21.15.102:0/856682858' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:49:19.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:19 smithi102 conmon[37544]: audit 2024-01-13T09:49:18.465347+0000 mon.smithi084 (mon.0) 344 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:49:19.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:19 smithi102 conmon[37544]: audit 2024-01-13T09:49:18.473463+0000 mon.smithi084 (mon.0) 345 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:49:19.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:19 smithi102 conmon[37544]: cluster 2024-01-13T09:49:18.473522+0000 mon.smithi084 (mon.0) 346 : cluster [DBG] osdmap e15: 0 total, 0 up, 0 in 2024-01-13T09:49:19.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:19 smithi102 conmon[37544]: audit 2024-01-13T09:49:18.784462+0000 mon.smithi084 (mon.0) 347 : audit [INF] from='client.? 172.21.15.84:0/730557136' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2d5bbd32-a6f8-4870-9778-7007667652bf"}]: dispatch 2024-01-13T09:49:19.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:19 smithi102 conmon[37544]: audit 2024-01- 2024-01-13T09:49:19.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:19 smithi102 conmon[37544]: 13T09:49:18.792735+0000 mon.smithi084 (mon.0) 348 : audit [INF] from='client.? 172.21.15.84:0/730557136' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2d5bbd32-a6f8-4870-9778-7007667652bf"}]': finished 2024-01-13T09:49:19.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:19 smithi102 conmon[37544]: cluster 2024-01-13T09:49:18.792793 2024-01-13T09:49:19.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:19 smithi102 conmon[37544]: +0000 mon.smithi084 (mon.0) 349 : cluster [DBG] osdmap e16: 1 total, 0 up, 1 in 2024-01-13T09:49:19.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:19 smithi102 conmon[37544]: audit 2024-01-13 2024-01-13T09:49:19.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:19 smithi102 conmon[37544]: T09:49:18.792888+0000 mon.smithi084 (mon.0 2024-01-13T09:49:19.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:19 smithi102 conmon[37544]: ) 350 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:49:19.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:19 smithi102 conmon[37544]: audit 2024-01-13T09:49: 2024-01-13T09:49:19.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:19 smithi102 conmon[37544]: 19.082585+0000 mon.smithi084 (mon.0) 351 : audit 2024-01-13T09:49:19.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:19 smithi102 conmon[37544]: [INF] from='client.? 172.21.15.84:0/2849679157' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:49:19.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:19 smithi102 conmon[37544]: audit 2024-01-13T09:49:19 2024-01-13T09:49:19.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:19 smithi102 conmon[37544]: .090622+0000 mon.smithi084 (mon.0) 352 : audit 2024-01-13T09:49:19.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:19 smithi102 conmon[37544]: [INF] from='client.? 172.21.15.84:0/2849679157' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:49:19.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:19 smithi102 conmon[37544]: cluster 2024-01-13T09:49:19. 2024-01-13T09:49:19.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:19 smithi102 conmon[37544]: 090662+0000 mon.smithi084 (mon.0) 353 : cluster [DBG] 2024-01-13T09:49:19.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:19 smithi102 conmon[37544]: osdmap e17: 0 total, 0 up, 0 in 2024-01-13T09:49:19.767 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:19 smithi084 conmon[33408]: audit 2024-01-13T09:49:18.465028+0000 mon.smithi102 (mon.1 2024-01-13T09:49:19.767 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:19 smithi084 conmon[33408]: ) 7 : audit [INF] from='client.? 172.21.15.102:0/856682858' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:49:19.767 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:19 smithi084 conmon[33408]: audit 2024-01-13T09:49:18.465347+0000 mon.smithi084 (mon.0) 344 : audit 2024-01-13T09:49:19.767 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:19 smithi084 conmon[33408]: [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:49:19.767 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:19 smithi084 conmon[33408]: audit 2024-01-13 2024-01-13T09:49:19.767 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:19 smithi084 conmon[33408]: T09:49:18.473463+0000 mon.smithi084 (mon.0) 345 : audit [INF] 2024-01-13T09:49:19.767 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:19 smithi084 conmon[33408]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:49:19.767 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:19 smithi084 conmon[33408]: cluster 2024-01-13T09:49:19.767 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:19 smithi084 conmon[33408]: 2024-01-13T09 2024-01-13T09:49:19.767 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:19 smithi084 conmon[33408]: :49:18.473522+0000 mon.smithi084 (mon.0) 346 : cluster [DBG] osdmap e15: 0 total, 0 up, 0 in 2024-01-13T09:49:19.767 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:19 smithi084 conmon[33408]: 2024-01-13T09:49:19.767 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:19 smithi084 conmon[33408]: audit 2024- 2024-01-13T09:49:19.768 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:19 smithi084 conmon[33408]: 01-13T09:49:18.784462+0000 mon.smithi084 2024-01-13T09:49:19.768 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:19 smithi084 conmon[33408]: (mon.0) 347 : audit [INF] from='client.? 172.21.15.84:0/730557136' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2d5bbd32-a6f8-4870-9778-7007667652bf"}]: dispatch 2024-01-13T09:49:19.768 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:19 smithi084 conmon[33408]: audit 2024 2024-01-13T09:49:19.768 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:19 smithi084 conmon[33408]: -01-13T09:49:18.792735+0000 mon.smithi084 ( 2024-01-13T09:49:19.768 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:19 smithi084 conmon[33408]: mon.0) 348 : audit [INF] from='client.? 172.21.15.84:0/730557136' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2d5bbd32-a6f8-4870-9778-7007667652bf"}]': finished 2024-01-13T09:49:19.768 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:19 smithi084 conmon[33408]: cluster 2024- 2024-01-13T09:49:19.768 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:19 smithi084 conmon[33408]: 01-13T09:49:18.792793+0000 mon.smithi084 ( 2024-01-13T09:49:19.768 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:19 smithi084 conmon[33408]: mon.0) 349 : cluster [DBG] osdmap e16: 1 total, 0 up, 1 in 2024-01-13T09:49:19.768 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:19 smithi084 conmon[33408]: audit 2024-01-13T09:49:19.768 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:19 smithi084 conmon[33408]: 2024-01-13T09:49:18.792888+0000 2024-01-13T09:49:19.768 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:19 smithi084 conmon[33408]: mon.smithi084 (mon.0) 350 2024-01-13T09:49:19.768 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:19 smithi084 conmon[33408]: : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:49:19.768 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:19 smithi084 conmon[33408]: audit 2024-01-13T 2024-01-13T09:49:19.769 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:19 smithi084 conmon[33408]: 09:49:19.082585+0000 mon.smithi084 (mon.0) 351 2024-01-13T09:49:19.769 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:19 smithi084 conmon[33408]: : audit [INF] from='client.? 172.21.15.84:0/2849679157' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:49:19.769 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:19 smithi084 conmon[33408]: audit 2024-01-13T09: 2024-01-13T09:49:19.769 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:19 smithi084 conmon[33408]: 49:19.090622+0000 mon.smithi084 (mon.0) 352 2024-01-13T09:49:19.769 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:19 smithi084 conmon[33408]: : audit [INF] from='client.? 172.21.15.84:0/2849679157' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:49:19.769 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:19 smithi084 conmon[33408]: cluster 2024-01-13T09: 2024-01-13T09:49:19.769 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:19 smithi084 conmon[33408]: 49:19.090662+0000 mon.smithi084 (mon.0) 353 2024-01-13T09:49:19.769 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:19 smithi084 conmon[33408]: : cluster [DBG] osdmap e17: 0 total, 0 up, 0 in 2024-01-13T09:49:20.123 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:49:20.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: cephadm 2024-01-13T09:49:19.187859+0000 mgr.smithi084.itrhok (mgr.14180) 121 : 2024-01-13T09:49:20.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: cephadm [ERR] executing create_from_spec_one(([('smithi084', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi102', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:49:20.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:49:20.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:49:20.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: return f(*arg) 2024-01-13T09:49:20.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:49:20.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:49:20.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:49:20.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: code, '\n'.join(err))) 2024-01-13T09:49:20.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:49:20.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4tfw79er:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpv188zisw:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:49:20.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:49:20.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:49:20.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:49:20.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 1a5d686c-e081-4e54-90c6-9f645c8b5710 2024-01-13T09:49:20.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:49:20.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:49:20.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:49:20.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:49:20.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:49:20.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr self.prepare() 2024-01-13T09:49:20.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:49:20.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:49:20.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:49:20.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:49:20.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:49:20.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:49:20.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:49:20.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:49:20.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:49:20.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:49:20.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:49:20.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:49:20.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:49:20.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:49:20.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:49:20.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:49:20.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:49:20.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:49:20.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:49:20.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:49:20.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:49:20.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:49:20.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:49:20.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:49:20.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:49:20.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:49:20.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:49:20.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:49:20.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:49:20.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:49:20.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:49:20.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:49:20.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:49:20.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:49:20.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:49:20.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:49:20.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:49:20.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:49:20.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:49:20.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:49:20.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:49:20.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:49:20.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:49:20.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:49:20.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:49:20.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:49:20.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:49:20.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:49:20.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:49:20.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:49:20.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:49:20.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:49:20.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: main() 2024-01-13T09:49:20.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:49:20.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: r = ctx.func(ctx) 2024-01-13T09:49:20.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:49:20.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:49:20.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:49:20.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:49:20.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:49:20.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:49:20.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90 2024-01-13T09:49:20.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: d2", line 1927, in _validate_fsid 2024-01-13T09:49:20.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:49:20.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:49:20.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:49:20.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:49:20.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:49:20.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4tfw79er:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpv188zisw:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:49:20.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: cluster 2024-01-13T09:49:19.565794+0000 mgr.smithi084.itrhok (mgr.14180) 122 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:20.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:20 smithi102 conmon[37544]: audit 2024-01-13T09:49:20.133779+0000 mon.smithi084 (mon.0) 354 : audit [DBG] from='client.? 172.21.15.84:0/3626641556' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:49:20.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: cephadm 2024-01-13T09:49:19.187859+0000 mgr.smithi084.itrhok (mgr.14180) 121 2024-01-13T09:49:20.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: : cephadm [ERR] executing create_from_spec_one(([('smithi084', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi102', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:49:20.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:49:20.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:49:20.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: return f(*arg) 2024-01-13T09:49:20.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:49:20.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:49:20.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:49:20.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: code, '\n'.join(err))) 2024-01-13T09:49:20.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:49:20.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4tfw79er:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpv188zisw:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:49:20.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:49:20.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:49:20.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:49:20.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 1a5d686c-e081-4e54-90c6-9f645c8b5710 2024-01-13T09:49:20.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:49:20.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:49:20.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:49:20.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:49:20.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:49:20.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr self.prepare() 2024-01-13T09:49:20.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:49:20.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:49:20.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:49:20.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:49:20.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:49:20.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:49:20.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:49:20.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:49:20.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:49:20.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:49:20.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:49:20.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:49:20.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:49:20.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:49:20.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:49:20.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:49:20.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:49:20.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:49:20.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:49:20.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:49:20.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:49:20.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:49:20.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:49:20.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:49:20.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:49:20.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:49:20.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:49:20.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:49:20.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:49:20.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:49:20.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:49:20.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:49:20.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:49:20.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:49:20.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:49:20.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:49:20.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:49:20.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:49:20.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:49:20.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:49:20.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:49:20.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:49:20.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:49:20.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:49:20.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:49:20.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:49:20.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:49:20.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:49:20.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:49:20.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:49:20.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:49:20.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:49:20.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: main() 2024-01-13T09:49:20.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:49:20.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: r = ctx.func(ctx) 2024-01-13T09:49:20.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:49:20.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:49:20.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:49:20.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:49:20.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:49:20.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:49:20.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83 2024-01-13T09:49:20.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: f90d2", line 1927, in _validate_fsid 2024-01-13T09:49:20.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:49:20.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:49:20.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:49:20.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:49:20.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:49:20.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4tfw79er:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpv188zisw:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:49:20.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: cluster 2024-01-13T09:49:19.565794+0000 mgr.smithi084.itrhok (mgr.14180) 122 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:20.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:20 smithi084 conmon[33408]: audit 2024-01-13T09:49:20.133779+0000 mon.smithi084 (mon.0) 354 : audit [DBG] from='client.? 172.21.15.84:0/3626641556' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:49:20.836 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:49:21.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: cephadm 2024-01-13T09:49:20.156069+0000 mgr.smithi084.itrhok (mgr.14180 2024-01-13T09:49:21.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: ) 123 : cephadm [ERR] executing create_from_spec_one(([('smithi084', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi102', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:49:21.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:49:21.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:49:21.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: return f(*arg) 2024-01-13T09:49:21.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:49:21.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:49:21.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:49:21.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: code, '\n'.join(err))) 2024-01-13T09:49:21.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi084/config 2024-01-13T09:49:21.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp5vfnafo2:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpfms4na3h:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:49:21.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:49:21.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:49:21.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:49:21.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 2d5bbd32-a6f8-4870-9778-7007667652bf 2024-01-13T09:49:21.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:49:21.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:49:21.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:49:21.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:49:21.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:49:21.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr self.prepare() 2024-01-13T09:49:21.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:49:21.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:49:21.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:49:21.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:49:21.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:49:21.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:49:21.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:49:21.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:49:21.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:49:21.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:49:21.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:49:21.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:49:21.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:49:21.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:49:21.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:49:21.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:49:21.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:49:21.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:49:21.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:49:21.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:49:21.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:49:21.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:49:21.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:49:21.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:49:21.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:49:21.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:49:21.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:49:21.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:49:21.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:49:21.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:49:21.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:49:21.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:49:21.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:49:21.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:49:21.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:49:21.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:49:21.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:49:21.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:49:21.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:49:21.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:49:21.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:49:21.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:49:21.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:49:21.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:49:21.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:49:21.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:49:21.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:49:21.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:49:21.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:49:21.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:49:21.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:49:21.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:49:21.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: main() 2024-01-13T09:49:21.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:49:21.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: r = ctx.func(ctx) 2024-01-13T09:49:21.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:49:21.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:49:21.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:49:21.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:49:21.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:49:21.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:49:21.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308 2024-01-13T09:49:21.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:49:21.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:49:21.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:49:21.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:49:21.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:49:21.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:49:21.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp5vfnafo2:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpfms4na3h:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:49:21.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: cephadm 2024-01-13T09:49:20.157760+0000 mgr.smithi084.itrhok (mgr.14180) 124 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T09:49:21.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: service_id: all-available-devices 2024-01-13T09:49:21.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: service_name: osd.all-available-devices 2024-01-13T09:49:21.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: placement: 2024-01-13T09:49:21.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: host_pattern: '*' 2024-01-13T09:49:21.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: spec: 2024-01-13T09:49:21.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: data_devices: 2024-01-13T09:49:21.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: all: true 2024-01-13T09:49:21.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: filter_logic: AND 2024-01-13T09:49:21.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: objectstore: bluestore 2024-01-13T09:49:21.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:49:21.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4tfw79er:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpv188zisw:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:49:21.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:49:21.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:49:21.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:49:21.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 1a5d686c-e081-4e54-90c6-9f645c8b5710 2024-01-13T09:49:21.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:49:21.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:49:21.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:49:21.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:49:21.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:49:21.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr self.prepare() 2024-01-13T09:49:21.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:49:21.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:49:21.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:49:21.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:49:21.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:49:21.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:49:21.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:49:21.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:49:21.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:49:21.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:49:21.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:49:21.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:49:21.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:49:21.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:49:21.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:49:21.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:49:21.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:49:21.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:49:21.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:49:21.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:49:21.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:49:21.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:49:21.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:49:21.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:49:21.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:49:21.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:49:21.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:49:21.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:49:21.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:49:21.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:49:21.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:49:21.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:49:21.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:49:21.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:49:21.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:49:21.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:49:21.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:49:21.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:49:21.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:49:21.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:49:21.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:49:21.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:49:21.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:49:21.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:49:21.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:49:21.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:49:21.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr devices = fin 2024-01-13T09:49:21.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: d_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:49:21.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:49:21.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:49:21.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:49:21.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:49:21.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:49:21.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: main() 2024-01-13T09:49:21.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:49:21.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: r = ctx.func(ctx) 2024-01-13T09:49:21.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:49:21.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:49:21.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:49:21.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:49:21.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:49:21.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:49:21.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:49:21.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:49:21.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:49:21.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:49:21.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:49:21.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:49:21.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4tfw79er:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpv188zisw:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:49:21.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:49:21.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T09:49:21.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: if self._apply_service(spec): 2024-01-13T09:49:21.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T09:49:21.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T09:49:21.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T09:49:21.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T09:49:21.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T09:49:21.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T09:49:21.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T09:49:21.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T09:49:21.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T09:49:21.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: raise self._value 2024-01-13T09:49:21.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T09:49:21.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: result = (True, func(*args, **kwds)) 2024-01-13T09:49:21.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T09:49:21.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: return list(map(*args)) 2024-01-13T09:49:21.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:49:21.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: return f(*arg) 2024-01-13T09:49:21.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:49:21.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:49:21.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:49:21.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: code, '\n'.join(err))) 2024-01-13T09:49:21.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:49:21.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4tfw79er:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpv188zisw:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:49:21.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:49:21.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:49:21.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:49:21.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 1a5d686c-e081-4e54-90c6-9f645c8b5710 2024-01-13T09:49:21.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:49:21.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:49:21.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:49:21.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:49:21.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:49:21.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr self.prepare() 2024-01-13T09:49:21.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:49:21.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:49:21.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:49:21.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:49:21.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:49:21.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:49:21.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:49:21.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:49:21.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:49:21.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:49:21.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:49:21.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:49:21.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:49:21.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:49:21.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:49:21.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:49:21.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:49:21.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", lin 2024-01-13T09:49:21.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: e 59, in newfunc 2024-01-13T09:49:21.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:49:21.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:49:21.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:49:21.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:49:21.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:49:21.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:49:21.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:49:21.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:49:21.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:49:21.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:49:21.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:49:21.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:49:21.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:49:21.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:49:21.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:49:21.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:49:21.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:49:21.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:49:21.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:49:21.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:49:21.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:49:21.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:49:21.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:49:21.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:49:21.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:49:21.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:49:21.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:49:21.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:49:21.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:49:21.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:49:21.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:49:21.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:49:21.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:49:21.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:49:21.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: main() 2024-01-13T09:49:21.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:49:21.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: r = ctx.func(ctx) 2024-01-13T09:49:21.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:49:21.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:49:21.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:49:21.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:49:21.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:49:21.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:49:21.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:49:21.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:49:21.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:49:21.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:49:21.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:49:21.756 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:49:21.756 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4tfw79er:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpv188zisw:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:49:21.756 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: cluster 2024-01-13T09:49:20.158549+0000 mgr.smithi084.itrhok (mgr.14180) 125 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:21.756 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:21 smithi102 conmon[37544]: cluster 2024-01-13T09:49:20.450411+0000 mon.smithi084 (mon.0) 355 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T09:49:21.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: cephadm 2024-01-13T09:49:20.156069+0000 mgr.smithi084.itrhok (mgr.14180) 123 : cephadm [ERR] executing create_from_spec_one(([('smithi084', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi102', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:49:21.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:49:21.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:49:21.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: return f(*arg) 2024-01-13T09:49:21.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:49:21.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:49:21.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:49:21.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: code, '\n'.join(err))) 2024-01-13T09:49:21.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi084/config 2024-01-13T09:49:21.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp5vfnafo2:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpfms4na3h:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:49:21.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:49:21.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:49:21.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:49:21.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 2d5bbd32-a6f8-4870-9778-7007667652bf 2024-01-13T09:49:21.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:49:21.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:49:21.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:49:21.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:49:21.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:49:21.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr self.prepare() 2024-01-13T09:49:21.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:49:21.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:49:21.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:49:21.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:49:21.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:49:21.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:49:21.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:49:21.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:49:21.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:49:21.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:49:21.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:49:21.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:49:21.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:49:21.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:49:21.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:49:21.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:49:21.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:49:21.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:49:21.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:49:21.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:49:21.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:49:21.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:49:21.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:49:21.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:49:21.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:49:21.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:49:21.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:49:21.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:49:21.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:49:21.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:49:21.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:49:21.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:49:21.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:49:21.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:49:21.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:49:21.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:49:21.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:49:21.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:49:21.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:49:21.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:49:21.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:49:21.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:49:21.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:49:21.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:49:21.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:49:21.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:49:21.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:49:21.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:49:21.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:49:21.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:49:21.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:49:21.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:49:21.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: main() 2024-01-13T09:49:21.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:49:21.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: r = ctx.func(ctx) 2024-01-13T09:49:21.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:49:21.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:49:21.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:49:21.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:49:21.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:49:21.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:49:21.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-8777 2024-01-13T09:49:21.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: 4f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:49:21.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:49:21.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:49:21.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:49:21.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:49:21.785 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:49:21.785 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp5vfnafo2:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpfms4na3h:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:49:21.785 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: cephadm 2024-01-13T09:49:20.157760+0000 mgr.smithi084.itrhok (mgr.14180) 124 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T09:49:21.785 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: service_id: all-available-devices 2024-01-13T09:49:21.785 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: service_name: osd.all-available-devices 2024-01-13T09:49:21.785 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: placement: 2024-01-13T09:49:21.785 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: host_pattern: '*' 2024-01-13T09:49:21.785 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: spec: 2024-01-13T09:49:21.785 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: data_devices: 2024-01-13T09:49:21.785 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: all: true 2024-01-13T09:49:21.785 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: filter_logic: AND 2024-01-13T09:49:21.785 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: objectstore: bluestore 2024-01-13T09:49:21.785 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:49:21.785 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4tfw79er:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpv188zisw:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:49:21.786 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:49:21.786 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:49:21.786 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:49:21.786 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 1a5d686c-e081-4e54-90c6-9f645c8b5710 2024-01-13T09:49:21.786 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:49:21.786 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:49:21.786 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:49:21.786 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:49:21.786 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:49:21.786 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr self.prepare() 2024-01-13T09:49:21.786 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:49:21.786 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:49:21.786 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:49:21.786 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:49:21.787 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:49:21.787 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:49:21.787 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:49:21.787 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:49:21.787 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:49:21.787 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:49:21.787 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:49:21.787 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:49:21.787 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:49:21.787 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:49:21.787 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:49:21.787 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:49:21.787 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:49:21.787 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:49:21.787 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:49:21.787 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:49:21.788 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:49:21.788 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:49:21.788 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:49:21.788 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:49:21.788 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:49:21.788 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:49:21.788 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:49:21.788 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:49:21.788 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:49:21.788 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:49:21.788 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:49:21.788 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:49:21.788 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:49:21.788 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:49:21.788 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:49:21.788 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:49:21.789 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:49:21.789 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:49:21.789 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:49:21.789 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:49:21.789 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:49:21.789 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:49:21.789 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:49:21.789 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:49:21.789 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:49:21.789 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/ 2024-01-13T09:49:21.789 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: lvm/zap.py", line 302, in zap_osd 2024-01-13T09:49:21.789 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:49:21.789 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:49:21.789 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:49:21.789 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:49:21.789 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:49:21.790 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:49:21.790 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: main() 2024-01-13T09:49:21.790 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:49:21.790 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: r = ctx.func(ctx) 2024-01-13T09:49:21.790 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:49:21.790 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:49:21.790 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:49:21.790 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:49:21.790 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:49:21.790 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:49:21.790 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:49:21.790 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:49:21.790 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:49:21.790 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:49:21.790 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:49:21.790 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:49:21.791 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4tfw79er:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpv188zisw:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:49:21.791 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:49:21.791 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T09:49:21.791 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: if self._apply_service(spec): 2024-01-13T09:49:21.791 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T09:49:21.791 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T09:49:21.791 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T09:49:21.791 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T09:49:21.791 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T09:49:21.791 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T09:49:21.791 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T09:49:21.791 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T09:49:21.791 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T09:49:21.791 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: raise self._value 2024-01-13T09:49:21.791 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T09:49:21.791 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: result = (True, func(*args, **kwds)) 2024-01-13T09:49:21.791 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T09:49:21.792 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: return list(map(*args)) 2024-01-13T09:49:21.792 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:49:21.792 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: return f(*arg) 2024-01-13T09:49:21.792 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:49:21.792 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:49:21.792 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:49:21.792 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: code, '\n'.join(err))) 2024-01-13T09:49:21.792 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:49:21.792 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4tfw79er:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpv188zisw:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:49:21.792 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:49:21.792 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:49:21.792 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:49:21.792 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 1a5d686c-e081-4e54-90c6-9f645c8b5710 2024-01-13T09:49:21.792 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:49:21.792 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:49:21.793 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:49:21.793 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:49:21.793 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:49:21.793 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr self.prepare() 2024-01-13T09:49:21.793 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:49:21.793 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:49:21.793 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:49:21.793 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:49:21.793 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:49:21.794 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:49:21.794 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:49:21.794 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:49:21.794 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:49:21.794 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:49:21.794 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:49:21.794 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:49:21.794 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:49:21.794 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:49:21.795 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:49:21.795 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:49:21.795 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:49:21.795 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:49:21.795 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:49:21.795 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:49:21.795 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:49:21.795 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:49:21.795 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:49:21.795 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:49:21.795 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:49:21.795 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:49:21.795 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:49:21.795 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:49:21.796 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:49:21.796 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:49:21.796 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:49:21.796 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:49:21.796 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:49:21.796 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:49:21.796 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:49:21.796 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:49:21.796 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:49:21.796 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:49:21.796 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:49:21.796 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:49:21.796 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:49:21.797 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:49:21.797 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:49:21.797 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:49:21.797 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:49:21.797 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:49:21.797 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:49:21.797 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:49:21.797 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:49:21.797 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:49:21.797 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:49:21.797 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:49:21.797 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:49:21.797 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: main() 2024-01-13T09:49:21.798 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:49:21.798 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: r = ctx.func(ctx) 2024-01-13T09:49:21.798 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:49:21.798 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:49:21.798 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:49:21.798 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:49:21.798 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:49:21.798 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:49:21.798 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:49:21.798 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:49:21.798 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:49:21.798 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:49:21.798 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:49:21.799 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:49:21.799 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4tfw79er:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpv188zisw:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:49:21.799 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: cluster 2024-01-13T09:49:20.158549+0000 mgr.smithi084.itrhok (mgr.14180) 125 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:21.799 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:21 smithi084 conmon[33408]: cluster 2024-01-13T09:49:20.450411+0000 mon.smithi084 (mon.0) 355 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T09:49:21.837 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:49:23.695 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:49:23.716 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:23 smithi084 conmon[33408]: cluster 2024-01-13T09:49:22.159021+0000 mgr.smithi084.itrhok (mgr.14180) 126 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:23.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:23 smithi102 conmon[37544]: cluster 2024-01-13T09:49:22.159021+0000 mgr.smithi084.itrhok (mgr.14180) 126 : cluster [DBG] 2024-01-13T09:49:23.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:23 smithi102 conmon[37544]: pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:24.294 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:49:24.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:24 smithi102 conmon[37544]: audit 2024-01-13T09:49:23.704676+0000 mon.smithi084 (mon.0) 356 : 2024-01-13T09:49:24.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:24 smithi102 conmon[37544]: audit [DBG] from='client.? 172.21.15.84:0/226034135' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:49:24.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:24 smithi084 conmon[33408]: audit 2024-01-13T09:49:23.704676+0000 mon.smithi084 (mon.0) 356 : audit [DBG] 2024-01-13T09:49:24.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:24 smithi084 conmon[33408]: from='client.? 172.21.15.84:0/226034135' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:49:25.295 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:49:25.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:25 smithi102 conmon[37544]: cluster 2024-01-13T09:49:24.159388+0000 mgr.smithi084.itrhok (mgr. 2024-01-13T09:49:25.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:25 smithi102 conmon[37544]: 14180) 127 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:25.752 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:25 smithi084 conmon[33408]: cluster 2024-01-13T09:49:24.159388+0000 mgr.smithi084.itrhok (mgr 2024-01-13T09:49:25.752 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:25 smithi084 conmon[33408]: .14180) 127 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:27.151 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:49:27.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:27 smithi102 conmon[37544]: cluster 2024-01-13T09:49:26.159795+0000 mgr.smithi084.itrhok (mgr.14180) 128 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:27.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:27 smithi102 conmon[37544]: 2024-01-13T09:49:27.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:27 smithi102 conmon[37544]: audit 2024-01-13T09:49:27.161247+0000 mon.smithi084 (mon.0) 357 : audit [DBG] from='client.? 172.21.15.84:0/957742527' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:49:27.764 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:27 smithi084 conmon[33408]: cluster 2024-01-13T09:49:26.159795+0000 mgr.smithi084.itrhok (mgr.14180) 128 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:27.764 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:27 smithi084 conmon[33408]: audit 2024-01-13T09:49:27.161247+0000 mon.smithi084 (mon.0) 357 : audit [DBG] from='client.? 172.21.15.84:0/957742527' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:49:27.783 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:49:28.785 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:49:29.686 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:29 smithi084 conmon[33408]: cluster 2024-01-13T09:49:28.160278+0000 mgr.smithi084.itrhok 2024-01-13T09:49:29.686 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:29 smithi084 conmon[33408]: (mgr.14180) 129 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:29.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:29 smithi102 conmon[37544]: cluster 2024-01-13T09:49:28.160278+0000 mgr.smithi084.itrhok (mgr.14180) 129 : cluster 2024-01-13T09:49:29.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:29 smithi102 conmon[37544]: [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:30.693 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:49:31.394 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:49:31.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:31 smithi102 conmon[37544]: cluster 2024-01-13T09:49:30.160907+0000 mgr.smithi084.itrhok (mgr.14180) 130 2024-01-13T09:49:31.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:31 smithi102 conmon[37544]: : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:31.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:31 smithi102 conmon[37544]: audit 2024-01-13T09:49:30.702603+0000 mon.smithi084 (mon.0) 358 : audit [DBG] from='client.? 172.21.15.84:0/4095183350' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:49:31.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:31 smithi084 conmon[33408]: cluster 2024-01-13T09:49:30.160907+0000 mgr.smithi084.itrhok (mgr.14180) 130 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:31.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:31 smithi084 conmon[33408]: audit 2024-01-13T09:49:30.702603+0000 mon.smithi084 (mon.0) 358 : audit [DBG] from='client.? 172.21.15.84:0/4095183350' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:49:32.394 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:49:33.527 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:33 smithi084 conmon[33408]: cluster 2024-01-13T09:49:32.161539+0000 mgr.smithi084.itrhok (mgr.14180) 131 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:33.734 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:33 smithi102 conmon[37544]: cluster 2024-01-13T09:49:32.161539+0000 mgr.smithi084.itrhok (mgr.14180) 131 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:34.332 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:49:34.660 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:34 smithi084 conmon[33408]: audit 2024-01-13T09:49:34.341012+0000 mon.smithi084 (mon.0 2024-01-13T09:49:34.660 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:34 smithi084 conmon[33408]: ) 359 : audit [DBG] from='client.? 172.21.15.84:0/3056168205' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:49:34.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:34 smithi102 conmon[37544]: audit 2024-01-13T09:49:34.341012+0000 mon.smithi084 (mon.0) 359 : 2024-01-13T09:49:34.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:34 smithi102 conmon[37544]: audit [DBG] from='client.? 172.21.15.84:0/3056168205' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:49:35.092 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:49:35.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:35 smithi084 conmon[33408]: cluster 2024-01-13T09:49:34.161929+0000 mgr.smithi084.itrhok (mgr.14180) 132 : cluster 2024-01-13T09:49:35.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:35 smithi084 conmon[33408]: [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:35.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:35 smithi102 conmon[37544]: cluster 2024-01-13T09:49:34.161929+0000 mgr.smithi084.itrhok (mgr.14180 2024-01-13T09:49:35.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:35 smithi102 conmon[37544]: ) 132 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:36.093 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:49:37.617 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:37 smithi084 conmon[33408]: cluster 2024-01-13T09:49:36.162352+0000 mgr.smithi084.itrhok (mgr 2024-01-13T09:49:37.617 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:37 smithi084 conmon[33408]: .14180) 133 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:37.962 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:49:37.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:37 smithi102 conmon[37544]: cluster 2024-01-13T09:49:36.162352+0000 mgr.smithi084.itrhok (mgr.14180) 133 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:37.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:37 smithi102 conmon[37544]: 2024-01-13T09:49:38.599 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:49:38.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:38 smithi084 conmon[33408]: audit 2024-01-13T09:49:37.971893+0000 mon.smithi084 (mon.0 2024-01-13T09:49:38.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:38 smithi084 conmon[33408]: ) 360 : audit [DBG] from='client.? 172.21.15.84:0/3954308982' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:49:38.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:38 smithi102 conmon[37544]: audit 2024-01-13T09:49:37.971893+0000 mon.smithi084 (mon.0) 360 : audit 2024-01-13T09:49:38.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:38 smithi102 conmon[37544]: [DBG] from='client.? 172.21.15.84:0/3954308982' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:49:39.600 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:49:39.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:39 smithi084 conmon[33408]: cluster 2024-01-13T09:49:38.162767+0000 mgr.smithi084.itrhok (mgr.14180) 134 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:39.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:39 smithi102 conmon[37544]: cluster 2024-01-13T09:49:38.162767+0000 mgr.smithi084.itrhok (mgr.14180 2024-01-13T09:49:39.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:39 smithi102 conmon[37544]: ) 134 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:41.460 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:49:41.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:41 smithi084 conmon[33408]: cluster 2024-01-13T09:49:40.163228+0000 mgr.smithi084.itrhok (mgr.14180) 135 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:41.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:41 smithi084 conmon[33408]: audit 2024-01-13T09:49:41.469650+0000 mon.smithi084 (mon.0) 361 : audit [DBG] from='client.? 172.21.15.84:0/713390718' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:49:41.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:41 smithi102 conmon[37544]: cluster 2024-01-13T09:49:40.163228+0000 mgr.smithi084.itrhok (mgr.14180) 135 : cluster [DBG] 2024-01-13T09:49:41.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:41 smithi102 conmon[37544]: pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:41.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:41 smithi102 conmon[37544]: audit 2024-01-13T09:49:41.469650+0000 mon.smithi084 (mon.0) 361 : audit [DBG] from='client.? 172.21.15.84:0/713390718' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:49:42.116 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:49:43.116 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:49:43.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:43 smithi084 conmon[33408]: cluster 2024-01-13T09:49:42.163762+0000 mgr.smithi084.itrhok (mgr.14180) 136 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:43.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:43 smithi102 conmon[37544]: cluster 2024-01-13T09:49:42.163762+0000 mgr.smithi084.itrhok (mgr.14180) 136 2024-01-13T09:49:43.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:43 smithi102 conmon[37544]: : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:45.000 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:49:45.681 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:49:45.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:45 smithi084 conmon[33408]: cluster 2024-01-13T09:49:44.164203+0000 mgr.smithi084.itrhok (mgr.14180) 137 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:45.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:45 smithi084 conmon[33408]: audit 2024-01-13T09:49:45.009285+0000 mon.smithi084 (mon.0) 362 : audit [DBG] from='client.? 172.21.15.84:0/1254924703' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:49:45.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:45 smithi102 conmon[37544]: cluster 2024-01-13T09:49:44.164203+0000 mgr.smithi084.itrhok (mgr.14180) 137 2024-01-13T09:49:45.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:45 smithi102 conmon[37544]: : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:45.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:45 smithi102 conmon[37544]: audit 2024-01-13T09:49:45.009285+0000 mon.smithi084 (mon.0) 362 : audit [DBG] from='client.? 172.21.15.84:0/1254924703' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:49:46.683 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:49:47.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:47 smithi084 conmon[33408]: cluster 2024-01-13T09:49:46.164686+0000 mgr.smithi084.itrhok (mgr.14180) 138 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:47.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:47 smithi102 conmon[37544]: cluster 2024-01-13T09:49:46.164686+0000 mgr.smithi084.itrhok (mgr.14180) 138 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:48.619 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:49:49.213 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:49:49.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:49 smithi084 conmon[33408]: cluster 2024-01-13T09:49:48.165035+0000 mgr.smithi084.itrhok (mgr.14180) 139 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:49.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:49 smithi084 conmon[33408]: audit 2024-01-13T09:49:48.628255+0000 mon.smithi084 (mon.0) 363 : audit [DBG] from='client.? 172.21.15.84:0/708233924' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:49:49.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:49 smithi102 conmon[37544]: cluster 2024-01-13T09:49:48.165035+0000 mgr.smithi084.itrhok (mgr.14180) 139 : cluster 2024-01-13T09:49:49.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:49 smithi102 conmon[37544]: [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:49.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:49 smithi102 conmon[37544]: audit 2024-01-13T09:49:48.628255+0000 mon.smithi084 (mon.0) 363 : audit [DBG] from='client.? 172.21.15.84:0/708233924' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:49:50.215 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:49:51.766 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:51 smithi084 conmon[33408]: cluster 2024-01-13T09:49:50.165580+0000 mgr.smithi084.itrhok (mgr.14180) 140 : cluster [DBG] 2024-01-13T09:49:51.766 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:51 smithi084 conmon[33408]: pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:51.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:51 smithi102 conmon[37544]: cluster 2024-01-13T09:49:50.165580+0000 mgr.smithi084.itrhok (mgr.14180) 140 : cluster 2024-01-13T09:49:51.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:51 smithi102 conmon[37544]: [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:52.152 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:49:52.870 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:52 smithi084 conmon[33408]: audit 2024-01-13T09:49:52.161912+0000 mon.smithi084 (mon.0) 364 : audit [DBG] from='client.? 172.21.15.84:0/674600083' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:49:52.883 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:49:52.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:52 smithi102 conmon[37544]: audit 2024-01-13T09:49:52.161912+0000 mon.smithi084 (mon.0) 364 : audit 2024-01-13T09:49:52.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:52 smithi102 conmon[37544]: [DBG] from='client.? 172.21.15.84:0/674600083' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:49:53.853 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:53 smithi084 conmon[33408]: cluster 2024-01-13T09:49:52.166062+0000 mgr.smithi084.itrhok (mgr.14180) 141 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:53.884 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:49:53.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:53 smithi102 conmon[37544]: cluster 2024-01-13T09:49:52.166062+0000 mgr.smithi084.itrhok (mgr.14180) 141 : cluster [DBG] 2024-01-13T09:49:53.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:53 smithi102 conmon[37544]: pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:55.784 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:49:55.807 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:55 smithi084 conmon[33408]: cluster 2024-01-13T09:49:54.166597+0000 mgr.smithi084.itrhok (mgr.14180) 142 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:55.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:55 smithi102 conmon[37544]: cluster 2024-01-13T09:49:54.166597+0000 mgr.smithi084.itrhok (mgr.14180) 142 : 2024-01-13T09:49:55.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:55 smithi102 conmon[37544]: cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:56.468 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:49:56.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:56 smithi084 conmon[33408]: audit 2024-01-13T09:49:55.792713+0000 mon.smithi084 (mon.0) 365 : audit [DBG] from='client.? 172.21.15.84:0/1954576608' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:49:56.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:56 smithi102 conmon[37544]: audit 2024-01-13T09:49:55.792713+0000 mon.smithi084 (mon.0) 365 : audit 2024-01-13T09:49:56.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:56 smithi102 conmon[37544]: [DBG] from='client.? 172.21.15.84:0/1954576608' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:49:57.535 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:49:57.984 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:57 smithi102 conmon[37544]: cluster 2024-01-13T09:49:56.166973+0000 mgr.smithi084.itrhok (mgr.14180) 143 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:57.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:57 smithi102 conmon[37544]: 2024-01-13T09:49:57.987 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:57 smithi084 conmon[33408]: cluster 2024-01-13T09:49:56.166973+0000 mgr.smithi084.itrhok (mgr.14180) 143 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:59.456 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:49:59.777 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:59 smithi084 conmon[33408]: cluster 2024-01-13T09:49:58.167635+0000 mgr.smithi084.itrhok (mgr.14180) 144 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:59.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:49:59 smithi084 conmon[33408]: audit 2024-01-13T09:49:59.465030+0000 mon.smithi084 (mon.0) 366 : audit [DBG] from='client.? 172.21.15.84:0/4294904676' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:49:59.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:59 smithi102 conmon[37544]: cluster 2024-01-13T09:49:58.167635+0000 mgr.smithi084.itrhok (mgr.14180) 144 : 2024-01-13T09:49:59.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:59 smithi102 conmon[37544]: cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:49:59.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:49:59 smithi102 conmon[37544]: audit 2024-01-13T09:49:59.465030+0000 mon.smithi084 (mon.0) 366 : audit [DBG] from='client.? 172.21.15.84:0/4294904676' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:50:00.115 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:50:00.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00.000165+0000 mon.smithi084 (mon.0) 367 : cluster [WRN] 2024-01-13T09:50:00.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: Health detail: HEALTH_WARN Failed to apply 1 service(s): osd.all-available-devices 2024-01-13T09:50:00.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00.000213+0000 mon.smithi084 (mon.0) 368 : cluster [WRN] [WRN] CEPHADM_APPLY_SPEC_FAIL: Failed to apply 1 service(s): osd.all-available-devices 2024-01-13T09:50:00.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00.000237+0000 mon.smithi084 (mon.0) 369 : cluster 2024-01-13T09:50:00.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: [WRN] osd.all-available-devices: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:50:00.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00 2024-01-13T09:50:00.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: .000291+0000 mon.smithi084 (mon.0) 370 : cluster [WRN] 2024-01-13T09:50:00.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4tfw79er:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpv188zisw:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:50:00.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00.000339+0000 mon.smithi084 (mon.0) 371 2024-01-13T09:50:00.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: : cluster [WRN] /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:50:00.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50 2024-01-13T09:50:00.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: :00.000361+0000 mon.smithi084 (mon.0) 372 : cluster 2024-01-13T09:50:00.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: [WRN] /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:50:00.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00 2024-01-13T09:50:00.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: .000382+0000 mon.smithi084 (mon.0) 373 : cluster [WRN] 2024-01-13T09:50:00.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:50:00.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00.000407 2024-01-13T09:50:00.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: +0000 mon.smithi084 (mon.0) 374 : cluster [WRN] /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 1a5d686c-e081-4e54-90c6-9f645c8b5710 2024-01-13T09:50:00.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 2024-01-13T09:50:00.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 2024-01-13T09:50:00.000438+0000 mon.smithi084 (mon.0 2024-01-13T09:50:00.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: ) 375 : cluster [WRN] /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:50:00.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T 2024-01-13T09:50:00.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 09:50:00.000471+0000 mon.smithi084 (mon.0) 2024-01-13T09:50:00.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 376 : cluster [WRN] /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:50:00.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09: 2024-01-13T09:50:00.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 50:00.000493+0000 mon.smithi084 (mon.0) 377 : 2024-01-13T09:50:00.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster [WRN] /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:50:00.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13 2024-01-13T09:50:00.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: T09:50:00.000514+0000 mon.smithi084 (mon.0) 2024-01-13T09:50:00.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 378 : cluster [WRN] /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:50:00.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T 2024-01-13T09:50:00.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 09:50:00.000538+0000 mon.smithi084 (mon.0) 2024-01-13T09:50:00.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 379 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:50:00.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00.000564+0000 2024-01-13T09:50:00.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: mon.smithi084 (mon.0) 380 : cluster [WRN] /bin/podman: stderr self.prepare() 2024-01-13T09:50:00.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 2024-01-13T09:50:00.000573+0000 mon.smithi084 ( 2024-01-13T09:50:00.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: mon.0) 381 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:00.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01 2024-01-13T09:50:00.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: -13T09:50:00.000580+0000 mon.smithi084 (mon.0 2024-01-13T09:50:00.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: ) 382 : cluster [WRN] /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:00.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09 2024-01-13T09:50:00.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: :50:00.000589+0000 mon.smithi084 (mon.0) 383 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:50:00.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024- 2024-01-13T09:50:00.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 01-13T09:50:00.000596+0000 mon.smithi084 (mon. 2024-01-13T09:50:00.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 0) 384 : cluster [WRN] /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:50:00.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13 2024-01-13T09:50:00.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: T09:50:00.000606+0000 mon.smithi084 (mon.0) 2024-01-13T09:50:00.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 385 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:50:00.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50 2024-01-13T09:50:00.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: :00.000615+0000 mon.smithi084 (mon.0) 386 : cluster [WRN] 2024-01-13T09:50:00.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:50:00.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00.000624+0000 2024-01-13T09:50:00.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: mon.smithi084 (mon.0) 387 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:50:00.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024- 2024-01-13T09:50:00.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 01-13T09:50:00.000633+0000 mon.smithi084 (mon.0) 2024-01-13T09:50:00.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 388 : cluster [WRN] /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:50:00.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09: 2024-01-13T09:50:00.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 50:00.000642+0000 mon.smithi084 (mon.0) 389 : cluster 2024-01-13T09:50:00.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: [WRN] /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:50:00.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00 2024-01-13T09:50:00.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: .000651+0000 mon.smithi084 (mon.0) 390 : cluster [WRN] 2024-01-13T09:50:00.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:50:00.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00.000660+0000 2024-01-13T09:50:00.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: mon.smithi084 (mon.0) 391 : cluster [WRN] /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:50:00.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024 2024-01-13T09:50:00.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: -01-13T09:50:00.000668+0000 mon.smithi084 (mon. 2024-01-13T09:50:00.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 0) 392 : cluster [WRN] /bin/podman: stderr 2024-01-13T09:50:00.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13 2024-01-13T09:50:00.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: T09:50:00.000679+0000 mon.smithi084 (mon.0) 393 2024-01-13T09:50:00.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: : cluster [WRN] /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:50:00.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09: 2024-01-13T09:50:00.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 50:00.000691+0000 mon.smithi084 (mon.0) 394 : cluster 2024-01-13T09:50:00.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: [WRN] /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:50:00.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00. 2024-01-13T09:50:00.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 000700+0000 mon.smithi084 (mon.0) 395 : cluster [WRN] 2024-01-13T09:50:00.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:50:00.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00.000710+0000 2024-01-13T09:50:00.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: mon.smithi084 (mon.0) 396 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:50:00.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024- 2024-01-13T09:50:00.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 01-13T09:50:00.000719+0000 mon.smithi084 (mon. 2024-01-13T09:50:00.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 0) 397 : cluster [WRN] /bin/podman: stderr self.main(self.argv) 2024-01-13T09:50:00.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01- 2024-01-13T09:50:00.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 13T09:50:00.000728+0000 mon.smithi084 (mon.0 2024-01-13T09:50:00.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: ) 398 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:50:00.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09: 2024-01-13T09:50:00.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 50:00.000736+0000 mon.smithi084 (mon.0) 399 : 2024-01-13T09:50:00.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster [WRN] /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:50:00.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00 2024-01-13T09:50:00.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: .000745+0000 mon.smithi084 (mon.0) 400 : cluster [WRN] 2024-01-13T09:50:00.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:50:00.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00.000754 2024-01-13T09:50:00.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: +0000 mon.smithi084 (mon.0) 401 : cluster [WRN] /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:50:00.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 2024-01-13T09:50:00.000761+0000 mon.smithi084 ( 2024-01-13T09:50:00.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: mon.0) 402 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:50:00.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T 2024-01-13T09:50:00.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 09:50:00.000770+0000 mon.smithi084 (mon.0) 403 : cluster 2024-01-13T09:50:00.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: [WRN] /bin/podman: stderr instance.main() 2024-01-13T09:50:00.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50: 2024-01-13T09:50:00.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 00.000778+0000 mon.smithi084 (mon.0) 404 : cluster [WRN] 2024-01-13T09:50:00.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:50:00.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00 2024-01-13T09:50:00.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: .000786+0000 mon.smithi084 (mon.0) 405 : cluster [WRN] /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:50:00.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 2024-01-13T09:50:00.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00.000797+0000 2024-01-13T09:50:00.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: mon.smithi084 (mon.0) 406 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:50:00.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 2024-01-13T09:50:00.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 2024-01-13T09:50:00.000805+0000 mon.smithi084 (mon.0) 407 2024-01-13T09:50:00.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: : cluster [WRN] /bin/podman: stderr instance.main() 2024-01-13T09:50:00.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09: 2024-01-13T09:50:00.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 50:00.000814+0000 mon.smithi084 (mon.0) 408 : cluster 2024-01-13T09:50:00.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:00.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50: 2024-01-13T09:50:00.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 00.000823+0000 mon.smithi084 (mon.0) 409 : cluster [WRN] 2024-01-13T09:50:00.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:00.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50: 2024-01-13T09:50:00.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 00.000832+0000 mon.smithi084 (mon.0) 410 : cluster 2024-01-13T09:50:00.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:50:00.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00.000844 2024-01-13T09:50:00.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: +0000 mon.smithi084 (mon.0) 411 : cluster [WRN] /bin/podman: stderr self._execute(plan) 2024-01-13T09:50:00.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 2024-01-13T09:50:00.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 2024-01-13T09:50:00.000860+0000 mon.smithi084 (mon.0) 2024-01-13T09:50:00.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 412 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:50:00.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09 2024-01-13T09:50:00.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: :50:00.000869+0000 mon.smithi084 (mon.0) 413 : 2024-01-13T09:50:00.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster [WRN] /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:50:00.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00. 2024-01-13T09:50:00.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 000878+0000 mon.smithi084 (mon.0) 414 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:00.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 2024-01-13T09:50:00.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00.000887+0000 mon.smithi084 2024-01-13T09:50:00.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: (mon.0) 415 : cluster [WRN] /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:00.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024- 2024-01-13T09:50:00.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 01-13T09:50:00.000896+0000 mon.smithi084 (mon.0 2024-01-13T09:50:00.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: ) 416 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:50:00.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09 2024-01-13T09:50:00.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: :50:00.000905+0000 mon.smithi084 (mon.0) 417 : cluster 2024-01-13T09:50:00.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: [WRN] /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:50:00.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00.000914 2024-01-13T09:50:00.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: +0000 mon.smithi084 (mon.0) 418 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:50:00.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 2024-01-13T09:50:00.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 2024-01-13T09:50:00.000923+0000 mon.smithi084 (mon.0 2024-01-13T09:50:00.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: ) 419 : cluster [WRN] /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:50:00.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09: 2024-01-13T09:50:00.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 50:00.000932+0000 mon.smithi084 (mon.0) 2024-01-13T09:50:00.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 420 : cluster [WRN] 2024-01-13T09:50:00.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:50:00.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00.000940 2024-01-13T09:50:00.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: +0000 mon.smithi084 (mon.0) 421 : cluster [WRN] /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:50:00.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 2024-01-13T09:50:00.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T 2024-01-13T09:50:00.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 09:50:00.000954+0000 mon.smithi084 (mon.0) 422 2024-01-13T09:50:00.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:50:00.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T 2024-01-13T09:50:00.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 09:50:00.000963+0000 mon.smithi084 (mon.0) 423 2024-01-13T09:50:00.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: : cluster [WRN] /bin/podman: stderr self.zap_osd() 2024-01-13T09:50:00.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09: 2024-01-13T09:50:00.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 50:00.000972+0000 mon.smithi084 (mon.0) 424 : cluster 2024-01-13T09:50:00.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:00.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024- 2024-01-13T09:50:00.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 01-13T09:50:00.000981+0000 mon.smithi084 (mon.0 2024-01-13T09:50:00.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: ) 425 : cluster [WRN] /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:00.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09 2024-01-13T09:50:00.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: :50:00.000991+0000 mon.smithi084 (mon.0) 426 : 2024-01-13T09:50:00.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:50:00.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50 2024-01-13T09:50:00.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: :00.000999+0000 mon.smithi084 (mon.0) 427 : cluster [WRN] 2024-01-13T09:50:00.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:50:00.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00.001009+0000 2024-01-13T09:50:00.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: mon.smithi084 (mon.0) 428 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:50:00.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 2024-01-13T09:50:00.001022+0000 mon.smithi084 (mon 2024-01-13T09:50:00.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: .0) 429 : cluster [WRN] /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:50:00.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01- 2024-01-13T09:50:00.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 13T09:50:00.001031+0000 mon.smithi084 (mon.0) 2024-01-13T09:50:00.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 430 : cluster [WRN] /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:50:00.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09 2024-01-13T09:50:00.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: :50:00.001039+0000 mon.smithi084 (mon.0) 431 : 2024-01-13T09:50:00.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster [WRN] Traceback (most recent call last): 2024-01-13T09:50:00.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50: 2024-01-13T09:50:00.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 00.001048+0000 mon.smithi084 (mon.0) 432 : cluster 2024-01-13T09:50:00.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: [WRN] File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:50:00.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00.001058 2024-01-13T09:50:00.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: +0000 mon.smithi084 (mon.0) 433 : cluster [WRN] main() 2024-01-13T09:50:00.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 2024-01-13T09:50:00.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 2024-01-13T09:50:00.001067+0000 mon.smithi084 (mon.0) 434 : cluster [WRN] File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:50:00.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 2024-01-13T09:50:00.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 2024-01-13T09:50:00.001077+0000 mon.smithi084 (mon.0) 2024-01-13T09:50:00.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 435 : cluster [WRN] r = ctx.func(ctx) 2024-01-13T09:50:00.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09 2024-01-13T09:50:00.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: :50:00.001086+0000 mon.smithi084 (mon.0) 436 : 2024-01-13T09:50:00.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster [WRN] File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:50:00.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50: 2024-01-13T09:50:00.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 00.001099+0000 mon.smithi084 (mon.0) 437 : cluster 2024-01-13T09:50:00.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: [WRN] return func(ctx) 2024-01-13T09:50:00.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00. 2024-01-13T09:50:00.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 001109+0000 mon.smithi084 (mon.0) 438 : cluster [WRN] 2024-01-13T09:50:00.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:50:01.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00.001118+0000 mon.smithi084 2024-01-13T09:50:01.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: (mon.0) 439 : cluster [WRN] return func(ctx) 2024-01-13T09:50:01.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024 2024-01-13T09:50:01.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: -01-13T09:50:00.001127+0000 mon.smithi084 (mon. 2024-01-13T09:50:01.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 0) 440 : cluster [WRN] File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:50:01.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01- 2024-01-13T09:50:01.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 13T09:50:00.001135+0000 mon.smithi084 (mon.0) 441 : 2024-01-13T09:50:01.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster [WRN] return func(ctx) 2024-01-13T09:50:01.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50 2024-01-13T09:50:01.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: :00.001143+0000 mon.smithi084 (mon.0) 442 : cluster 2024-01-13T09:50:01.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: [WRN] File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:50:01.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50: 2024-01-13T09:50:01.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 00.001152+0000 mon.smithi084 (mon.0) 443 : cluster 2024-01-13T09:50:01.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: [WRN] return func(ctx) 2024-01-13T09:50:01.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00.001160 2024-01-13T09:50:01.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: +0000 mon.smithi084 (mon.0) 444 : cluster [WRN] 2024-01-13T09:50:01.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:50:01.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00.001170+0000 mon.smithi084 (mon.0) 445 : cluster 2024-01-13T09:50:01.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: [WRN] out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:50:01.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:01.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 2024-01-13T09:50:00.001179+0000 mon.smithi084 (mon.0) 446 2024-01-13T09:50:01.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: : cluster [WRN] File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:50:01.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50: 2024-01-13T09:50:01.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 00.001191+0000 mon.smithi084 (mon.0) 447 : cluster 2024-01-13T09:50:01.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: [WRN] raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:50:01.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00. 2024-01-13T09:50:01.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: 001200+0000 mon.smithi084 (mon.0) 448 : cluster [WRN] 2024-01-13T09:50:01.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:00 smithi102 conmon[37544]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4tfw79er:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpv188zisw:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:50:01.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:00.000165+0000 mon.smithi084 (mon.0) 367 : cluster [WRN] Health detail: HEALTH_WARN Failed to apply 1 service(s): osd.all-available-devices 2024-01-13T09:50:01.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:00.000213+0000 2024-01-13T09:50:01.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: mon.smithi084 (mon.0) 368 : cluster [WRN] [WRN] CEPHADM_APPLY_SPEC_FAIL: Failed to apply 1 service(s): osd.all-available-devices 2024-01-13T09:50:01.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:00.000237+0000 mon.smithi084 (mon.0) 369 : cluster [WRN] osd.all-available-devices: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:50:01.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:00.000291+0000 mon.smithi084 (mon.0) 370 : cluster [WRN] Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4tfw79er:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpv188zisw:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:50:01.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:00.000339+0000 mon.smithi084 (mon.0) 371 : cluster [WRN] /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:50:01.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:00.000361+0000 mon.smithi084 (mon.0) 372 : cluster [WRN] /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:50:01.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:00.000382+0000 mon.smithi084 (mon.0) 373 : cluster [WRN] /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:50:01.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:00.000407+0000 mon.smithi084 (mon.0) 374 : cluster [WRN] /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 1a5d686c-e081-4e54-90c6-9f645c8b5710 2024-01-13T09:50:01.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:00.000438+0000 mon.smithi084 (mon.0) 375 : cluster [WRN] /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:50:01.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:00.000471+0000 mon.smithi084 (mon.0) 376 : cluster [WRN] /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:50:01.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:00.000493+0000 mon.smithi084 (mon.0) 377 : cluster [WRN] /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:50:01.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:00.000514+0000 mon.smithi084 (mon.0) 378 : cluster [WRN] /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:50:01.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:00.000538+0000 mon.smithi084 ( 2024-01-13T09:50:01.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: mon.0) 379 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:50:01.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:00.000564+0000 mon.smithi084 (mon.0) 380 : cluster [WRN] /bin/podman: stderr self.prepare() 2024-01-13T09:50:01.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024 2024-01-13T09:50:01.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: -01-13T09:50:00.000573+0000 mon.smithi084 (mon.0) 381 : cluster [WRN] 2024-01-13T09:50:01.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:01.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024- 2024-01-13T09:50:01.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 01-13T09:50:00.000580+0000 mon.smithi084 (mon.0) 382 2024-01-13T09:50:01.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: : cluster [WRN] /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:01.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01- 2024-01-13T09:50:01.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 13T09:50:00.000589+0000 mon.smithi084 (mon.0) 383 2024-01-13T09:50:01.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:50:01.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:00. 2024-01-13T09:50:01.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 000596+0000 mon.smithi084 (mon.0) 384 : cluster [WRN] 2024-01-13T09:50:01.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:50:01.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:01.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 2024-01-13T09:50:00.000606+0000 mon.smithi084 (mon.0) 385 2024-01-13T09:50:01.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:50:01.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:00 2024-01-13T09:50:01.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: .000615+0000 mon.smithi084 (mon.0) 386 : cluster 2024-01-13T09:50:01.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: [WRN] /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:50:01.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:01.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 2024-01-13T09:50:00.000624+0000 mon.smithi084 (mon.0 2024-01-13T09:50:01.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: ) 387 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:50:01.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50: 2024-01-13T09:50:01.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 00.000633+0000 mon.smithi084 (mon.0) 388 : cluster 2024-01-13T09:50:01.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: [WRN] /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:50:01.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:01.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 2024-01-13T09:50:00.000642+0000 mon.smithi084 (mon. 2024-01-13T09:50:01.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 0) 389 : cluster [WRN] /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:50:01.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50 2024-01-13T09:50:01.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: :00.000651+0000 mon.smithi084 (mon.0) 390 : 2024-01-13T09:50:01.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster [WRN] /bin/podman: stderr 2024-01-13T09:50:01.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:01.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 2024-01-13T09:50:00.000660+0000 mon.smithi084 ( 2024-01-13T09:50:01.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: mon.0) 391 : cluster [WRN] /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:50:01.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09 2024-01-13T09:50:01.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: :50:00.000668+0000 mon.smithi084 (mon.0) 2024-01-13T09:50:01.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 392 : cluster [WRN] /bin/podman: stderr 2024-01-13T09:50:01.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50: 2024-01-13T09:50:01.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 00.000679+0000 mon.smithi084 (mon.0) 393 : cluster 2024-01-13T09:50:01.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: [WRN] /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:50:01.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:01.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 2024-01-13T09:50:00.000691+0000 mon.smithi084 (mon. 2024-01-13T09:50:01.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 0) 394 : cluster [WRN] /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:50:01.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09: 2024-01-13T09:50:01.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 50:00.000700+0000 mon.smithi084 (mon.0) 395 : 2024-01-13T09:50:01.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster [WRN] /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:50:01.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:01.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 2024-01-13T09:50:00.000710+0000 mon.smithi084 (mon 2024-01-13T09:50:01.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: .0) 396 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:50:01.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50 2024-01-13T09:50:01.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: :00.000719+0000 mon.smithi084 (mon.0) 397 : 2024-01-13T09:50:01.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster [WRN] /bin/podman: stderr self.main(self.argv) 2024-01-13T09:50:01.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:00.000728+0000 2024-01-13T09:50:01.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: mon.smithi084 (mon.0) 398 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:50:01.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:00 2024-01-13T09:50:01.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: .000736+0000 mon.smithi084 (mon.0) 399 : cluster 2024-01-13T09:50:01.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: [WRN] /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:50:01.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:01.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 2024-01-13T09:50:00.000745+0000 mon.smithi084 (mon. 2024-01-13T09:50:01.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 0) 400 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:50:01.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50 2024-01-13T09:50:01.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: :00.000754+0000 mon.smithi084 (mon.0) 401 : 2024-01-13T09:50:01.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster [WRN] /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:50:01.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:00. 2024-01-13T09:50:01.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 000761+0000 mon.smithi084 (mon.0) 402 : cluster [WRN] 2024-01-13T09:50:01.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:50:01.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024 2024-01-13T09:50:01.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: -01-13T09:50:00.000770+0000 mon.smithi084 (mon.0 2024-01-13T09:50:01.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: ) 403 : cluster [WRN] /bin/podman: stderr instance.main() 2024-01-13T09:50:01.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50 2024-01-13T09:50:01.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: :00.000778+0000 mon.smithi084 (mon.0) 404 : 2024-01-13T09:50:01.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:50:01.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:01.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 2024-01-13T09:50:00.000786 2024-01-13T09:50:01.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: +0000 mon.smithi084 (mon.0) 405 : cluster [WRN] 2024-01-13T09:50:01.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:50:01.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:01.036 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 2024-01-13T09:50:00.000797+0000 mon.smithi084 (mon.0 2024-01-13T09:50:01.036 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: ) 406 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:50:01.036 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50 2024-01-13T09:50:01.036 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: :00.000805+0000 mon.smithi084 (mon.0) 407 : cluster 2024-01-13T09:50:01.036 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: [WRN] /bin/podman: stderr instance.main() 2024-01-13T09:50:01.036 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:01.036 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 2024-01-13T09:50:00.000814+0000 mon.smithi084 ( 2024-01-13T09:50:01.036 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: mon.0) 408 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:01.036 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T 2024-01-13T09:50:01.036 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 09:50:00.000823+0000 mon.smithi084 (mon.0 2024-01-13T09:50:01.036 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: ) 409 : cluster [WRN] /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:01.036 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50 2024-01-13T09:50:01.036 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: :00.000832+0000 mon.smithi084 (mon.0) 410 : 2024-01-13T09:50:01.036 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:50:01.036 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:00. 2024-01-13T09:50:01.037 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 000844+0000 mon.smithi084 (mon.0) 411 : cluster [WRN] 2024-01-13T09:50:01.037 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:50:01.037 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:01.037 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 2024-01-13T09:50:00.000860+0000 mon.smithi084 (mon.0 2024-01-13T09:50:01.037 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: ) 412 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:50:01.037 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09: 2024-01-13T09:50:01.037 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 50:00.000869+0000 mon.smithi084 (mon.0) 413 2024-01-13T09:50:01.037 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: : cluster [WRN] /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:50:01.037 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:00. 2024-01-13T09:50:01.037 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 000878+0000 mon.smithi084 (mon.0) 414 : cluster 2024-01-13T09:50:01.037 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:01.037 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:00 2024-01-13T09:50:01.037 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: .000887+0000 mon.smithi084 (mon.0) 415 : cluster [WRN] 2024-01-13T09:50:01.037 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:01.037 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024 2024-01-13T09:50:01.038 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: -01-13T09:50:00.000896+0000 mon.smithi084 (mon.0) 2024-01-13T09:50:01.038 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 416 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:50:01.038 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:00 2024-01-13T09:50:01.038 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: .000905+0000 mon.smithi084 (mon.0) 417 : cluster 2024-01-13T09:50:01.038 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: [WRN] /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:50:01.038 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:01.038 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 2024-01-13T09:50:00.000914+0000 mon.smithi084 (mon. 2024-01-13T09:50:01.038 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 0) 418 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:50:01.038 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01- 2024-01-13T09:50:01.038 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 13T09:50:00.000923+0000 mon.smithi084 (mon.0 2024-01-13T09:50:01.038 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: ) 419 : cluster [WRN] /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:50:01.038 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50 2024-01-13T09:50:01.038 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: :00.000932+0000 mon.smithi084 (mon.0) 420 : 2024-01-13T09:50:01.038 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:50:01.038 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:00 2024-01-13T09:50:01.038 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: .000940+0000 mon.smithi084 (mon.0) 421 : cluster 2024-01-13T09:50:01.039 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: [WRN] /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:50:01.039 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:01.039 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 2024-01-13T09:50:00.000954+0000 mon.smithi084 (mon.0 2024-01-13T09:50:01.039 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: ) 422 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:50:01.039 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50 2024-01-13T09:50:01.039 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: :00.000963+0000 mon.smithi084 (mon.0) 423 : 2024-01-13T09:50:01.039 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster [WRN] /bin/podman: stderr self.zap_osd() 2024-01-13T09:50:01.039 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:00 2024-01-13T09:50:01.039 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: .000972+0000 mon.smithi084 (mon.0) 424 : cluster 2024-01-13T09:50:01.039 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:01.039 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:01.039 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 2024-01-13T09:50:00.000981+0000 mon.smithi084 (mon 2024-01-13T09:50:01.039 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: .0) 425 : cluster [WRN] /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:01.039 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09 2024-01-13T09:50:01.039 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: :50:00.000991+0000 mon.smithi084 (mon.0) 2024-01-13T09:50:01.040 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 426 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:50:01.040 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:00. 2024-01-13T09:50:01.040 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 000999+0000 mon.smithi084 (mon.0) 427 : cluster [WRN] 2024-01-13T09:50:01.040 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:50:01.040 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024- 2024-01-13T09:50:01.040 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 01-13T09:50:00.001009+0000 mon.smithi084 (mon.0) 2024-01-13T09:50:01.040 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 428 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:50:01.040 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:00 2024-01-13T09:50:01.040 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: .001022+0000 mon.smithi084 (mon.0) 429 : cluster [WRN] 2024-01-13T09:50:01.040 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:50:01.040 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T 2024-01-13T09:50:01.040 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 09:50:00.001031+0000 mon.smithi084 (mon.0) 430 : cluster [WRN] 2024-01-13T09:50:01.040 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:50:01.040 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024- 2024-01-13T09:50:01.040 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 01-13T09:50:00.001039+0000 mon.smithi084 (mon.0) 2024-01-13T09:50:01.041 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 431 : cluster [WRN] Traceback (most recent call last): 2024-01-13T09:50:01.041 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:00 2024-01-13T09:50:01.041 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: .001048+0000 mon.smithi084 (mon.0) 432 : cluster 2024-01-13T09:50:01.041 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: [WRN] File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:50:01.041 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:01.041 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 2024-01-13T09:50:00.001058+0000 mon.smithi084 (mon. 2024-01-13T09:50:01.041 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 0) 433 : cluster [WRN] main() 2024-01-13T09:50:01.041 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50 2024-01-13T09:50:01.041 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: :00.001067+0000 mon.smithi084 (mon.0) 434 : 2024-01-13T09:50:01.041 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster [WRN] File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:50:01.041 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:00. 2024-01-13T09:50:01.041 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 001077+0000 mon.smithi084 (mon.0) 435 : cluster [WRN] 2024-01-13T09:50:01.041 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: r = ctx.func(ctx) 2024-01-13T09:50:01.041 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024 2024-01-13T09:50:01.041 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: -01-13T09:50:00.001086+0000 mon.smithi084 (mon. 2024-01-13T09:50:01.042 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 0) 436 : cluster [WRN] File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:50:01.042 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:00.001099+0000 mon.smithi084 ( 2024-01-13T09:50:01.042 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: mon.0) 437 : cluster [WRN] return func(ctx) 2024-01-13T09:50:01.042 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09: 2024-01-13T09:50:01.042 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 50:00.001109+0000 mon.smithi084 (mon.0) 438 : 2024-01-13T09:50:01.042 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster [WRN] File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:50:01.042 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:01.042 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 2024-01-13T09:50:00.001118+0000 mon.smithi084 (mon 2024-01-13T09:50:01.042 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: .0) 439 : cluster [WRN] return func(ctx) 2024-01-13T09:50:01.042 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09 2024-01-13T09:50:01.042 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: :50:00.001127+0000 mon.smithi084 (mon.0) 440 : cluster 2024-01-13T09:50:01.042 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: [WRN] File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:50:01.042 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024 2024-01-13T09:50:01.042 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: -01-13T09:50:00.001135+0000 mon.smithi084 (mon.0) 2024-01-13T09:50:01.042 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 441 : cluster [WRN] return func(ctx) 2024-01-13T09:50:01.043 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09 2024-01-13T09:50:01.043 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: :50:00.001143+0000 mon.smithi084 (mon.0) 2024-01-13T09:50:01.043 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 442 : cluster [WRN] File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:50:01.043 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:00 2024-01-13T09:50:01.043 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: .001152+0000 mon.smithi084 (mon.0) 443 : cluster 2024-01-13T09:50:01.043 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: [WRN] return func(ctx) 2024-01-13T09:50:01.043 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:01.043 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 2024-01-13T09:50:00.001160+0000 mon.smithi084 (mon. 2024-01-13T09:50:01.043 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 0) 444 : cluster [WRN] File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:50:01.043 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T 2024-01-13T09:50:01.043 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 09:50:00.001170+0000 mon.smithi084 (mon.0 2024-01-13T09:50:01.043 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: ) 445 : cluster [WRN] out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:50:01.043 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50 2024-01-13T09:50:01.043 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: :00.001179+0000 mon.smithi084 (mon.0) 446 : 2024-01-13T09:50:01.043 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster [WRN] File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:50:01.044 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:01.044 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 2024-01-13T09:50:00.001191+0000 mon.smithi084 (mon 2024-01-13T09:50:01.044 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: .0) 447 : cluster [WRN] raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:50:01.044 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: cluster 2024-01-13T 2024-01-13T09:50:01.044 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: 09:50:00.001200+0000 mon.smithi084 (mon.0 2024-01-13T09:50:01.044 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:00 smithi084 conmon[33408]: ) 448 : cluster [WRN] RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4tfw79er:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpv188zisw:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:50:01.117 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:50:01.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:01 smithi102 conmon[37544]: cluster 2024-01-13T09:50:00.167972+0000 mgr.smithi084.itrhok (mgr.14180) 2024-01-13T09:50:01.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:01 smithi102 conmon[37544]: 145 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:02.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:01 smithi084 conmon[33408]: cluster 2024-01-13T09:50:00.167972+0000 mgr.smithi084.itrhok (mgr.14180) 145 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:03.148 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:50:03.744 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:50:03.854 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:03 smithi084 conmon[33408]: cluster 2024-01-13T09:50:02.168555+0000 mgr.smithi084.itrhok (mgr.14180) 146 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:03.855 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:03 smithi084 conmon[33408]: audit 2024-01-13T09:50:03.157567+0000 mon.smithi084 (mon.0) 449 : audit [DBG] from='client.? 172.21.15.84:0/1008926443' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:50:03.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:03 smithi102 conmon[37544]: cluster 2024-01-13T09:50:02.168555+0000 mgr.smithi084.itrhok (mgr.14180) 146 : cluster 2024-01-13T09:50:03.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:03 smithi102 conmon[37544]: [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:03.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:03 smithi102 conmon[37544]: audit 2024-01-13T09:50:03.157567+0000 mon.smithi084 (mon.0) 449 : audit [DBG] from='client.? 172.21.15.84:0/1008926443' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:50:04.745 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:50:05.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:05 smithi102 conmon[37544]: cluster 2024-01-13T09:50:04.168973+0000 mgr.smithi084.itrhok (mgr.14180) 147 : 2024-01-13T09:50:05.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:05 smithi102 conmon[37544]: cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:06.027 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:05 smithi084 conmon[33408]: cluster 2024-01-13T09:50:04.168973+0000 mgr.smithi084.itrhok (mgr.14180) 147 : cluster 2024-01-13T09:50:06.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:05 smithi084 conmon[33408]: [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:06.641 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:50:07.307 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:50:07.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:07 smithi102 conmon[37544]: cluster 2024-01-13T09:50:06.169390+0000 mgr.smithi084.itrhok (mgr.14180) 148 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:07.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:07 smithi102 conmon[37544]: audit 2024-01-13T09:50:07.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:07 smithi102 conmon[37544]: 2024-01-13T09:50:06.649480+0000 mon.smithi084 (mon.0) 450 : audit [DBG] from='client.? 172.21.15.84:0/319692319' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:50:08.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:07 smithi084 conmon[33408]: cluster 2024-01-13T09:50:06.169390+0000 mgr.smithi084.itrhok (mgr.14180) 148 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:08.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:07 smithi084 conmon[33408]: audit 2024-01-13T09:50:06.649480+0000 mon.smithi084 (mon.0) 450 : audit [DBG] from='client.? 172.21.15.84:0/319692319' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:50:08.308 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:50:09.887 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:09 smithi084 conmon[33408]: cluster 2024-01-13T09:50:08.169846+0000 mgr.smithi084.itrhok (mgr.14180) 149 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:09.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:09 smithi102 conmon[37544]: cluster 2024-01-13T09:50:08.169846+0000 mgr.smithi084.itrhok (mgr.14180) 149 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:10.251 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:50:10.937 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:10 smithi084 conmon[33408]: audit 2024-01-13T09:50:10.260072+0000 mon.smithi084 (mon.0) 451 : audit [DBG] from='client.? 172.21.15.84:0/3727266324' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:50:10.951 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:50:10.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:10 smithi102 conmon[37544]: audit 2024-01-13T09:50:10.260072+0000 mon.smithi084 (mon.0) 451 2024-01-13T09:50:10.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:10 smithi102 conmon[37544]: : audit [DBG] from='client.? 172.21.15.84:0/3727266324' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:50:11.952 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:50:11.984 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:11 smithi084 conmon[33408]: cluster 2024-01-13T09:50:10.170362+0000 mgr.smithi084.itrhok (mgr.14180) 150 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:11.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:11 smithi102 conmon[37544]: cluster 2024-01-13T09:50:10.170362+0000 mgr.smithi084.itrhok (mgr.14180) 150 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:13.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:13 smithi084 conmon[33408]: cluster 2024-01-13T09:50:12.170768+0000 mgr.smithi084.itrhok (mgr.14180) 151 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:13.909 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:50:13.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:13 smithi102 conmon[37544]: cluster 2024-01-13T09:50:12.170768+0000 mgr.smithi084.itrhok (mgr.14180) 151 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:14.526 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:50:14.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:14 smithi084 conmon[33408]: audit 2024-01-13T09:50:13.917744+0000 mon.smithi084 (mon.0) 452 : audit [DBG] from='client.? 172.21.15.84:0/1945308694' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:50:14.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:14 smithi102 conmon[37544]: audit 2024-01-13T09:50:13.917744+0000 mon.smithi084 (mon.0) 452 : audit [DBG] from='client.? 172.21.15.84:0/1945308694' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:50:15.527 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:50:15.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:15 smithi102 conmon[37544]: cluster 2024-01-13T09:50:14.171143+0000 mgr.smithi084.itrhok (mgr.14180) 2024-01-13T09:50:15.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:15 smithi102 conmon[37544]: 152 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:16.008 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:15 smithi084 conmon[33408]: cluster 2024-01-13T09:50:14.171143+0000 mgr.smithi084.itrhok (mgr.14180 2024-01-13T09:50:16.009 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:15 smithi084 conmon[33408]: ) 152 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:17.502 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:50:17.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:17 smithi102 conmon[37544]: cluster 2024-01-13T09:50:16.171676+0000 mgr.smithi084.itrhok (mgr.14180) 153 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:17.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:17 smithi102 conmon[37544]: audit 2024-01-13T09:50:17.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:17 smithi102 conmon[37544]: 2024-01-13T09:50:17.510416+0000 mon.smithi084 (mon.0) 453 : audit [DBG] from='client.? 172.21.15.84:0/284702809' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:50:18.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:17 smithi084 conmon[33408]: cluster 2024-01-13T09:50:16.171676+0000 mgr.smithi084.itrhok (mgr.14180) 153 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:18.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:17 smithi084 conmon[33408]: audit 2024-01-13T09:50:17.510416+0000 mon.smithi084 (mon.0) 453 : audit [DBG] from='client.? 172.21.15.84:0/284702809' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:50:18.186 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:50:19.187 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:50:19.863 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:19 smithi084 conmon[33408]: cluster 2024-01-13T09:50:18.172221+0000 mgr.smithi084.itrhok (mgr.14180) 154 : cluster [DBG] 2024-01-13T09:50:19.863 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:19 smithi084 conmon[33408]: pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:19.975 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:19 smithi102 conmon[37544]: cluster 2024-01-13T09:50:18.172221+0000 mgr.smithi084.itrhok (mgr.14180) 154 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:20.828 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:20 smithi084 conmon[33408]: audit 2024-01-13T09:50:20.160000+0000 mon.smithi084 (mon.0 2024-01-13T09:50:20.828 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:20 smithi084 conmon[33408]: ) 454 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T09:50:20.828 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:20 smithi084 conmon[33408]: audit 2024-01-13T09:50:20.568489+0000 mon.smithi084 (mon.0) 455 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:50:20.829 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:20 smithi084 conmon[33408]: audit 2024-01-13T09:50:20.569157+0000 mon.smithi084 (mon.0) 456 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T09:50:20.829 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:20 smithi084 conmon[33408]: audit 2024-01-13T09:50:20.574007+0000 mon.smithi084 (mon.0) 457 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:50:20.829 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:20 smithi084 conmon[33408]: audit 2024-01-13T09:50:20.579614+0000 mon.smithi084 (mon.0) 458 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T09:50:20.829 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:20 smithi084 conmon[33408]: audit 2024-01-13T09:50:20.583226+0000 mon.smithi084 (mon.0) 459 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:50:20.829 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:20 smithi084 conmon[33408]: audit 2024-01-13T09:50 2024-01-13T09:50:20.829 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:20 smithi084 conmon[33408]: :20.584303+0000 mon.smithi084 (mon.0) 460 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:50:20.829 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:20 smithi084 conmon[33408]: audit 2024-01-13T09:50:20.584532+0000 mon.smithi084 (mon.0) 461 : audit [DBG] 2024-01-13T09:50:20.829 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:20 smithi084 conmon[33408]: from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:50:20.829 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:20 smithi084 conmon[33408]: audit 2024-01-13T09 2024-01-13T09:50:20.829 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:20 smithi084 conmon[33408]: :50:20.585017+0000 mon.smithi084 (mon.0) 462 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:50:20.829 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:20 smithi084 conmon[33408]: 2024-01-13T09:50:20.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:20 smithi102 conmon[37544]: audit 2024-01-13T09:50:20.160000+0000 mon.smithi084 (mon.0) 454 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T09:50:20.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:20 smithi102 conmon[37544]: audit 2024-01-13T09:50:20.568489+0000 mon.smithi084 (mon.0) 455 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:50:20.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:20 smithi102 conmon[37544]: audit 2024-01-13T09:50:20.569157+0000 mon.smithi084 (mon.0) 456 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T09:50:20.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:20 smithi102 conmon[37544]: audit 2024-01-13T09:50:20.574007+0000 mon.smithi084 (mon.0) 457 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:50:20.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:20 smithi102 conmon[37544]: audit 2024-01-13T09:50:20.579614+0000 mon.smithi084 (mon.0) 458 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T09:50:20.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:20 smithi102 conmon[37544]: audit 2024-01-13T09:50:20.583226+0000 mon.smithi084 (mon.0) 459 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:50:20.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:20 smithi102 conmon[37544]: audit 2024-01-13T09:50:20.584303+0000 mon.smithi084 (mon.0) 460 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:50:20.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:20 smithi102 conmon[37544]: audit 2024-01-13T09:50:20.584532+0000 mon.smithi084 (mon.0) 461 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:50:20.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:20 smithi102 conmon[37544]: audit 2024-01-13T09:50:20.585017+0000 mon.smithi084 (mon.0) 462 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:50:21.210 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:50:21.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:21 smithi102 conmon[37544]: cluster 2024-01-13T09:50:20.172625+0000 mgr.smithi084.itrhok (mgr.14180) 155 : cluster 2024-01-13T09:50:21.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:21 smithi102 conmon[37544]: [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:21.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:21 smithi102 conmon[37544]: cluster 2024-01-13T09:50:20.574488+0000 mgr.smithi084.itrhok (mgr.14180) 156 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:21.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:21 smithi102 conmon[37544]: cluster 2024-01-13T09:50:20.615576+0000 mon.smithi084 (mon.0) 463 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T09:50:21.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:21 smithi102 conmon[37544]: cluster 2024-01-13T09:50:20.615615+0000 mon.smithi084 (mon.0) 464 : cluster [INF] Cluster is now healthy 2024-01-13T09:50:21.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:21 smithi102 conmon[37544]: audit 2024-01-13T09:50:21.218860+0000 mon.smithi084 (mon.0) 465 : audit [DBG] from='client.? 172.21.15.84:0/2821912179' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:50:21.895 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:21 smithi084 conmon[33408]: cluster 2024-01-13T09:50:20.172625+0000 mgr.smithi084.itrhok (mgr.14180) 155 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:21.895 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:21 smithi084 conmon[33408]: cluster 2024-01-13T09:50:20.574488+0000 mgr.smithi084.itrhok (mgr.14180) 156 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:21.895 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:21 smithi084 conmon[33408]: cluster 2024-01-13T09:50:20.615576+0000 mon.smithi084 (mon.0) 463 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T09:50:21.895 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:21 smithi084 conmon[33408]: cluster 2024-01-13T09:50:20.615615+0000 mon.smithi084 (mon.0) 464 : cluster [INF] Cluster is now healthy 2024-01-13T09:50:21.895 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:21 smithi084 conmon[33408]: audit 2024-01-13T09:50:21.218860+0000 mon.smithi084 (mon.0) 465 : audit [DBG] from='client.? 172.21.15.84:0/2821912179' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:50:22.130 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:50:23.132 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:50:23.855 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:23 smithi084 conmon[33408]: cluster 2024-01-13T09:50:22.574779+0000 mgr.smithi084.itrhok (mgr.14180) 157 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:23.855 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:23 smithi084 conmon[33408]: audit 2024-01-13T09:50:23.123157+0000 mon.smithi102 (mon.1) 8 : audit [INF] from='client.? 172.21.15.102:0/2204340350' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82bb432f-24c0-40d1-8173-c57cd5b133ef"}]: dispatch 2024-01-13T09:50:23.856 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:23 smithi084 conmon[33408]: audit 2024-01-13T09:50:23.123383+0000 mon.smithi084 (mon.0) 466 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82bb432f-24c0-40d1-8173-c57cd5b133ef"}]: dispatch 2024-01-13T09:50:23.856 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:23 smithi084 conmon[33408]: audit 2024-01-13T09:50:23.127528+0000 mon.smithi084 (mon.0) 467 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "82bb432f-24c0-40d1-8173-c57cd5b133ef"}]': finished 2024-01-13T09:50:23.856 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:23 smithi084 conmon[33408]: cluster 2024-01-13T09:50:23.127578+0000 mon.smithi084 (mon.0) 468 : cluster [DBG] osdmap e18: 1 total, 0 up, 1 in 2024-01-13T09:50:23.856 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:23 smithi084 conmon[33408]: audit 2024-01-13T09:50:23.127683+0000 mon.smithi084 (mon.0) 469 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:50:23.856 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:23 smithi084 conmon[33408]: audit 2024-01-13T09:50:23.411721+0000 mon.smithi102 (mon.1) 9 : audit [INF] from='client.? 172.21.15.102:0/1842661134' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:50:23.856 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:23 smithi084 conmon[33408]: audit 2024-01-13T09:50:23.412148+0000 mon.smithi084 (mon.0) 470 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:50:23.856 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:23 smithi084 conmon[33408]: audit 2024-01-13 2024-01-13T09:50:23.856 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:23 smithi084 conmon[33408]: T09:50:23.416717+0000 mon.smithi084 (mon.0) 471 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:50:23.856 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:23 smithi084 conmon[33408]: cluster 2024-01-13T09:50:23.416791+0000 mon.smithi084 (mon.0) 472 : cluster [DBG] osdmap e19: 0 total, 0 up, 0 in 2024-01-13T09:50:23.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:23 smithi102 conmon[37544]: cluster 2024-01-13T09:50:22.574779+0000 mgr.smithi084.itrhok (mgr 2024-01-13T09:50:23.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:23 smithi102 conmon[37544]: .14180) 157 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:23.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:23 smithi102 conmon[37544]: audit 2024-01-13T09:50:23.123157+0000 mon.smithi102 (mon.1) 8 : audit [INF] from='client.? 172.21.15.102:0/2204340350' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82bb432f-24c0-40d1-8173-c57cd5b133ef"}]: dispatch 2024-01-13T09:50:23.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:23 smithi102 conmon[37544]: audit 2024-01-13T09:50:23.123383+0000 mon.smithi084 (mon.0) 466 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82bb432f-24c0-40d1-8173-c57cd5b133ef"}]: dispatch 2024-01-13T09:50:23.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:23 smithi102 conmon[37544]: audit 2024-01-13T09:50:23.127528+0000 mon.smithi084 (mon.0) 467 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "82bb432f-24c0-40d1-8173-c57cd5b133ef"}]': finished 2024-01-13T09:50:23.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:23 smithi102 conmon[37544]: cluster 2024-01-13T09:50:23.127578+0000 mon.smithi084 (mon.0) 468 : cluster [DBG] osdmap e18: 1 total, 0 up, 1 in 2024-01-13T09:50:23.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:23 smithi102 conmon[37544]: audit 2024-01- 2024-01-13T09:50:23.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:23 smithi102 conmon[37544]: 13T09:50:23.127683+0000 mon.smithi084 (mon.0) 469 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:50:23.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:23 smithi102 conmon[37544]: audit 2024-01-13T09:50:23 2024-01-13T09:50:23.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:23 smithi102 conmon[37544]: .411721+0000 mon.smithi102 (mon.1) 9 : audit [INF] from='client.? 172.21.15.102:0/1842661134' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:50:23.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:23 smithi102 conmon[37544]: audit 2024-01-13T09:50:23.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:23 smithi102 conmon[37544]: 2024-01-13T09:50 2024-01-13T09:50:23.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:23 smithi102 conmon[37544]: :23.412148+0000 mon.smithi084 (mon.0) 2024-01-13T09:50:23.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:23 smithi102 conmon[37544]: 470 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:50:23.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:23 smithi102 conmon[37544]: audit 2024-01- 2024-01-13T09:50:23.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:23 smithi102 conmon[37544]: 13T09:50:23.416717+0000 mon.smithi084 (mon 2024-01-13T09:50:23.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:23 smithi102 conmon[37544]: .0) 471 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:50:23.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:23 smithi102 conmon[37544]: cluster 2024 2024-01-13T09:50:23.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:23 smithi102 conmon[37544]: -01-13T09:50: 2024-01-13T09:50:23.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:23 smithi102 conmon[37544]: 23.416791+0000 mon.smithi084 (mon.0 2024-01-13T09:50:23.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:23 smithi102 conmon[37544]: ) 472 : cluster [DBG] osdmap e19: 0 total, 0 up, 0 in 2024-01-13T09:50:24.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:24 smithi084 conmon[33408]: audit 2024-01-13T09:50:24.056781+0000 mon.smithi084 ( 2024-01-13T09:50:24.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:24 smithi084 conmon[33408]: mon.0) 473 : audit [INF] from='client.? 172.21.15.84:0/4142749878' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c6c6ab9-4584-4cec-8279-f09e10e03d97"}]: dispatch 2024-01-13T09:50:24.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:24 smithi084 conmon[33408]: audit 2024-01-13T09:50:24.060179+0000 mon.smithi084 (mon.0) 474 : audit [INF] from='client.? 172.21.15.84:0/4142749878' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7c6c6ab9-4584-4cec-8279-f09e10e03d97"}]': finished 2024-01-13T09:50:24.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:24 smithi084 conmon[33408]: cluster 2024-01-13T09:50:24.060217+0000 mon.smithi084 (mon.0) 475 : cluster [DBG] osdmap e20: 1 total, 0 up, 1 in 2024-01-13T09:50:24.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:24 smithi084 conmon[33408]: audit 2024-01-13T09:50:24.060314+0000 mon.smithi084 (mon.0) 476 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:50:24.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:24 smithi084 conmon[33408]: audit 2024-01-13T09:50:24.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:24 smithi084 conmon[33408]: 2024-01-13T09:50:24.353025+0000 mon.smithi084 (mon.0) 477 : audit [INF] from='client.? 172.21.15.84:0/4122863461' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:50:24.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:24 smithi084 conmon[33408]: audit 2024-01- 2024-01-13T09:50:24.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:24 smithi084 conmon[33408]: 13T09:50:24.356780+0000 mon.smithi084 (mon.0) 478 : audit 2024-01-13T09:50:24.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:24 smithi084 conmon[33408]: [INF] from='client.? 172.21.15.84:0/4122863461' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:50:24.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:24 smithi084 conmon[33408]: cluster 2024-01-13T09:50:24.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:24 smithi084 conmon[33408]: 2024-01-13T09:50:24.356822+0000 mon.smithi084 (mon. 2024-01-13T09:50:24.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:24 smithi084 conmon[33408]: 0) 479 : cluster [DBG] osdmap e21: 0 total, 0 up, 0 in 2024-01-13T09:50:24.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:24 smithi102 conmon[37544]: audit 2024-01-13T09:50:24.056781+0000 mon.smithi084 (mon.0) 473 : audit [INF] from='client.? 172.21.15.84:0/4142749878' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c6c6ab9-4584-4cec-8279-f09e10e03d97"}]: dispatch 2024-01-13T09:50:24.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:24 smithi102 conmon[37544]: audit 2024-01-13 2024-01-13T09:50:24.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:24 smithi102 conmon[37544]: T09:50:24.060179+0000 mon.smithi084 (mon.0) 474 : audit [INF] from='client.? 172.21.15.84:0/4142749878' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7c6c6ab9-4584-4cec-8279-f09e10e03d97"}]': finished 2024-01-13T09:50:24.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:24 smithi102 conmon[37544]: cluster 2024-01-13T09:50:24.060217+0000 mon.smithi084 (mon.0) 475 : cluster [DBG] osdmap e20: 1 total, 0 up, 1 in 2024-01-13T09:50:24.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:24 smithi102 conmon[37544]: audit 2024-01-13T09:50:24.060314+0000 mon.smithi084 (mon.0) 476 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:50:24.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:24 smithi102 conmon[37544]: audit 2024-01-13T09:50:24.353025+0000 mon.smithi084 (mon.0) 477 : audit [INF] from='client.? 172.21.15.84:0/4122863461' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:50:24.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:24 smithi102 conmon[37544]: audit 2024-01-13T09:50:24.356780+0000 mon.smithi084 (mon.0) 478 : audit [INF] from='client.? 172.21.15.84:0/4122863461' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:50:24.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:24 smithi102 conmon[37544]: cluster 2024-01-13T09:50:24.356822+0000 mon.smithi084 (mon.0) 479 : cluster [DBG] osdmap e21: 0 total, 0 up, 0 in 2024-01-13T09:50:25.607 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:50:25.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: cephadm 2024-01-13T09:50:24.095328+0000 mgr.smithi084.itrhok ( 2024-01-13T09:50:25.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: mgr.14180) 158 : cephadm [ERR] executing create_from_spec_one(([('smithi084', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi102', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:50:25.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:50:25.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:50:25.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: return f(*arg) 2024-01-13T09:50:25.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:50:25.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:50:25.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:50:25.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: code, '\n'.join(err))) 2024-01-13T09:50:25.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:50:25.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4t8rzoeb:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpn_kkvg16:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:50:25.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:50:25.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:50:25.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:50:25.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 82bb432f-24c0-40d1-8173-c57cd5b133ef 2024-01-13T09:50:25.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:50:25.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:50:25.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:50:25.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:50:25.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:50:25.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr self.prepare() 2024-01-13T09:50:25.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:25.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:25.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:50:25.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:50:25.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:50:25.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:50:25.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:50:25.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:50:25.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:50:25.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:50:25.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:50:25.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:50:25.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:50:25.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:50:25.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:50:25.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:50:25.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:50:25.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:50:25.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:50:25.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:50:25.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:50:25.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:50:25.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:50:25.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:50:25.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:50:25.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:50:25.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:50:25.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:25.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:25.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:50:25.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:50:25.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:50:25.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:50:25.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:25.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:25.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:50:25.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:50:25.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:50:25.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:50:25.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:50:25.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:50:25.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:50:25.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:50:25.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:25.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:25.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:50:25.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:50:25.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:50:25.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:50:25.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:50:25.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:50:25.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:50:25.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: main() 2024-01-13T09:50:25.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:50:25.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: r = ctx.func(ctx) 2024-01-13T09:50:25.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:50:25.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:50:25.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:50:25.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:50:25.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:50:25.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:50:25.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39fe 2024-01-13T09:50:25.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: bbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:50:25.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:50:25.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:50:25.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:50:25.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:50:25.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:50:25.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4t8rzoeb:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpn_kkvg16:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:50:25.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: cluster 2024-01-13T09:50:24.575097+0000 mgr.smithi084.itrhok (mgr.14180) 159 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:25.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: cluster 2024-01-13T09:50:25.354972+0000 mon.smithi084 (mon.0) 480 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T09:50:25.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:25 smithi102 conmon[37544]: audit 2024-01-13T09:50:25.616402+0000 mon.smithi084 (mon.0) 481 : audit [DBG] from='client.? 172.21.15.84:0/2718237041' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:50:26.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: cephadm 2024-01-13T09:50:24.095328+0000 mgr.smithi084.itrhok (mgr.14180) 2024-01-13T09:50:26.028 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: 158 : cephadm [ERR] executing create_from_spec_one(([('smithi084', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi102', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:50:26.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:50:26.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:50:26.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: return f(*arg) 2024-01-13T09:50:26.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:50:26.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:50:26.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:50:26.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: code, '\n'.join(err))) 2024-01-13T09:50:26.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:50:26.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4t8rzoeb:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpn_kkvg16:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:50:26.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:50:26.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:50:26.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:50:26.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 82bb432f-24c0-40d1-8173-c57cd5b133ef 2024-01-13T09:50:26.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:50:26.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:50:26.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:50:26.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:50:26.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:50:26.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr self.prepare() 2024-01-13T09:50:26.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:26.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:26.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:50:26.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:50:26.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:50:26.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:50:26.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:50:26.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:50:26.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:50:26.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:50:26.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:50:26.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:50:26.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:50:26.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:50:26.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:50:26.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:50:26.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:50:26.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:50:26.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:50:26.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:50:26.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:50:26.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:50:26.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:50:26.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:50:26.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:50:26.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:50:26.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:50:26.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:26.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:26.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:50:26.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:50:26.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:50:26.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:50:26.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:26.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:26.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:50:26.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:50:26.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:50:26.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:50:26.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:50:26.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:50:26.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:50:26.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:50:26.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:26.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:26.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:50:26.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:50:26.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:50:26.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:50:26.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:50:26.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:50:26.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:50:26.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: main() 2024-01-13T09:50:26.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:50:26.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: r = ctx.func(ctx) 2024-01-13T09:50:26.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:50:26.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:50:26.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:50:26.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:50:26.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:50:26.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:50:26.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fd 2024-01-13T09:50:26.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: a83f90d2", line 1927, in _validate_fsid 2024-01-13T09:50:26.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:50:26.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:50:26.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:50:26.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:50:26.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:50:26.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4t8rzoeb:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpn_kkvg16:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:50:26.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: cluster 2024-01-13T09:50:24.575097+0000 mgr.smithi084.itrhok (mgr.14180) 159 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:26.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: cluster 2024-01-13T09:50:25.354972+0000 mon.smithi084 (mon.0) 480 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T09:50:26.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:25 smithi084 conmon[33408]: audit 2024-01-13T09:50:25.616402+0000 mon.smithi084 (mon.0) 481 : audit [DBG] from='client.? 172.21.15.84:0/2718237041' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:50:26.276 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:50:26.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: cephadm 2024-01-13T09:50:25.275080+0000 mgr.smithi084.itrhok (mgr.14180) 160 : cephadm [ERR] executing create_from_spec_one(([('smithi084', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi102', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:50:26.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:50:26.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:50:26.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: return f(*arg) 2024-01-13T09:50:26.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:50:26.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:50:26.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:50:26.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: code, '\n'.join(err))) 2024-01-13T09:50:26.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi084/config 2024-01-13T09:50:26.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp7jiosqcx:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpny5fsem2:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:50:26.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:50:26.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:50:26.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:50:26.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 7c6c6ab9-4584-4cec-8279-f09e10e03d97 2024-01-13T09:50:26.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:50:26.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:50:26.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:50:26.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:50:26.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:50:26.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr self.prepare() 2024-01-13T09:50:26.987 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:26.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:26.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:50:26.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:50:26.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:50:26.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:50:26.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:50:26.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:50:26.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:50:26.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:50:26.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:50:26.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:50:26.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:50:26.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:50:26.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:50:26.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:50:26.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:50:26.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:50:26.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:50:26.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:50:26.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:50:26.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:50:26.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:50:26.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:50:26.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:50:26.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:50:26.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:50:26.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:26.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:26.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:50:26.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:50:26.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:50:26.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:50:26.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:26.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:26.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:50:26.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:50:26.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:50:26.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:50:26.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:50:26.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:50:26.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:50:26.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:50:26.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:26.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:26.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:50:26.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:50:26.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:50:26.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:50:26.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:50:26.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:50:26.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:50:26.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: main() 2024-01-13T09:50:26.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:50:26.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: r = ctx.func(ctx) 2024-01-13T09:50:26.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:50:26.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:50:26.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:50:26.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:50:26.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:50:26.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:50:26.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-8777 2024-01-13T09:50:26.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: 4f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:50:26.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:50:26.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:50:26.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:50:26.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:50:26.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:50:26.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp7jiosqcx:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpny5fsem2:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:50:26.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: cephadm 2024-01-13T09:50:25.276211+0000 mgr.smithi084.itrhok (mgr.14180) 161 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T09:50:26.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: service_id: all-available-devices 2024-01-13T09:50:26.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: service_name: osd.all-available-devices 2024-01-13T09:50:26.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: placement: 2024-01-13T09:50:26.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: host_pattern: '*' 2024-01-13T09:50:26.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: spec: 2024-01-13T09:50:26.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: data_devices: 2024-01-13T09:50:26.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: all: true 2024-01-13T09:50:26.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: filter_logic: AND 2024-01-13T09:50:26.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: objectstore: bluestore 2024-01-13T09:50:26.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:50:26.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4t8rzoeb:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpn_kkvg16:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:50:26.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:50:26.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:50:26.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:50:26.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 82bb432f-24c0-40d1-8173-c57cd5b133ef 2024-01-13T09:50:26.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:50:26.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:50:26.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:50:26.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:50:26.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:50:26.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr self.prepare() 2024-01-13T09:50:26.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:26.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:26.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:50:26.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:50:26.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:50:26.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:50:26.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:50:26.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:50:26.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:50:26.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:50:26.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:50:26.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:50:26.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:50:26.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:50:26.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:50:26.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:50:26.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:50:26.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:50:26.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:50:26.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:50:26.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:50:26.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:50:26.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:50:26.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:50:26.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:50:26.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:50:26.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:50:26.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:26.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:26.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:50:26.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:50:26.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:50:26.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:50:26.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:26.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:26.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:50:26.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:50:26.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:50:26.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:50:26.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:50:26.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:50:26.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:50:26.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:50:26.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:26.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:26.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/ 2024-01-13T09:50:26.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: lvm/zap.py", line 302, in zap_osd 2024-01-13T09:50:26.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:50:26.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:50:26.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:50:26.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:50:26.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:50:26.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:50:26.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: main() 2024-01-13T09:50:26.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:50:26.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: r = ctx.func(ctx) 2024-01-13T09:50:26.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:50:26.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:50:26.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:50:26.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:50:26.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:50:26.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:50:26.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:50:26.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:50:26.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:50:27.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:50:27.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:50:27.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:50:27.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4t8rzoeb:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpn_kkvg16:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:50:27.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:50:27.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T09:50:27.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: if self._apply_service(spec): 2024-01-13T09:50:27.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T09:50:27.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T09:50:27.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T09:50:27.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T09:50:27.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T09:50:27.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T09:50:27.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T09:50:27.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T09:50:27.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T09:50:27.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: raise self._value 2024-01-13T09:50:27.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T09:50:27.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: result = (True, func(*args, **kwds)) 2024-01-13T09:50:27.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T09:50:27.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: return list(map(*args)) 2024-01-13T09:50:27.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:50:27.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: return f(*arg) 2024-01-13T09:50:27.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:50:27.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:50:27.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:50:27.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: code, '\n'.join(err))) 2024-01-13T09:50:27.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:50:27.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4t8rzoeb:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpn_kkvg16:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:50:27.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:50:27.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:50:27.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:50:27.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 82bb432f-24c0-40d1-8173-c57cd5b133ef 2024-01-13T09:50:27.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:50:27.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:50:27.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:50:27.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:50:27.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:50:27.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr self.prepare() 2024-01-13T09:50:27.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:27.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:27.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:50:27.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:50:27.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:50:27.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:50:27.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:50:27.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:50:27.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:50:27.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:50:27.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:50:27.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:50:27.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:50:27.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:50:27.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:50:27.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:50:27.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:50:27.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:50:27.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:50:27.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:50:27.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:50:27.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:50:27.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:50:27.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:50:27.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:50:27.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:50:27.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:50:27.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:50:27.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:27.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:27.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:50:27.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:50:27.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:50:27.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:50:27.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:27.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:27.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:50:27.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:50:27.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:50:27.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:50:27.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:50:27.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:50:27.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:50:27.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:50:27.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:27.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:27.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:50:27.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:50:27.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:50:27.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:50:27.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:50:27.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:50:27.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:50:27.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: main() 2024-01-13T09:50:27.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:50:27.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: r = ctx.func(ctx) 2024-01-13T09:50:27.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:50:27.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:50:27.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:50:27.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:50:27.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:50:27.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:50:27.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:50:27.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:50:27.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:50:27.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:50:27.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:50:27.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:50:27.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4t8rzoeb:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpn_kkvg16:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:50:27.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:26 smithi102 conmon[37544]: cluster 2024-01-13T09:50:25.276711+0000 mgr.smithi084.itrhok (mgr.14180) 162 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:27.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: cephadm 2024-01-13T09:50:25.275080+0000 mgr.smithi084.itrhok (mgr.14180) 160 : cephadm [ERR] executing create_from_spec_one(([('smithi084', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi102', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:50:27.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:50:27.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:50:27.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: return f(*arg) 2024-01-13T09:50:27.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:50:27.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:50:27.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:50:27.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: code, '\n'.join(err))) 2024-01-13T09:50:27.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi084/config 2024-01-13T09:50:27.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp7jiosqcx:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpny5fsem2:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:50:27.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:50:27.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:50:27.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:50:27.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 7c6c6ab9-4584-4cec-8279-f09e10e03d97 2024-01-13T09:50:27.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:50:27.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:50:27.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:50:27.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:50:27.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:50:27.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr self.prepare() 2024-01-13T09:50:27.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:27.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:27.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:50:27.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:50:27.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:50:27.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:50:27.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:50:27.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:50:27.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:50:27.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:50:27.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:50:27.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:50:27.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:50:27.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:50:27.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:50:27.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:50:27.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:50:27.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:50:27.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:50:27.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:50:27.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:50:27.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:50:27.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:50:27.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:50:27.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:50:27.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:50:27.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:50:27.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:27.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:27.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:50:27.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:50:27.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:50:27.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:50:27.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:27.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:27.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:50:27.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:50:27.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:50:27.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:50:27.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:50:27.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:50:27.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:50:27.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:50:27.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:27.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:27.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:50:27.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:50:27.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:50:27.033 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:50:27.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:50:27.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:50:27.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:50:27.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: main() 2024-01-13T09:50:27.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:50:27.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: r = ctx.func(ctx) 2024-01-13T09:50:27.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:50:27.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:50:27.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:50:27.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:50:27.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:50:27.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:50:27.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-8777 2024-01-13T09:50:27.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: 4f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:50:27.034 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:50:27.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:50:27.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:50:27.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:50:27.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:50:27.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp7jiosqcx:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpny5fsem2:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:50:27.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: cephadm 2024-01-13T09:50:25.276211+0000 mgr.smithi084.itrhok (mgr.14180) 161 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T09:50:27.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: service_id: all-available-devices 2024-01-13T09:50:27.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: service_name: osd.all-available-devices 2024-01-13T09:50:27.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: placement: 2024-01-13T09:50:27.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: host_pattern: '*' 2024-01-13T09:50:27.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: spec: 2024-01-13T09:50:27.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: data_devices: 2024-01-13T09:50:27.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: all: true 2024-01-13T09:50:27.035 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: filter_logic: AND 2024-01-13T09:50:27.036 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: objectstore: bluestore 2024-01-13T09:50:27.036 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:50:27.036 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4t8rzoeb:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpn_kkvg16:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:50:27.036 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:50:27.036 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:50:27.036 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:50:27.037 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 82bb432f-24c0-40d1-8173-c57cd5b133ef 2024-01-13T09:50:27.037 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:50:27.037 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:50:27.037 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:50:27.037 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:50:27.037 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:50:27.037 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr self.prepare() 2024-01-13T09:50:27.037 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:27.038 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:27.038 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:50:27.038 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:50:27.038 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:50:27.038 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:50:27.038 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:50:27.038 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:50:27.038 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:50:27.038 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:50:27.038 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:50:27.038 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:50:27.038 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:50:27.038 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:50:27.038 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:50:27.038 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:50:27.039 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:50:27.039 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:50:27.039 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:50:27.039 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:50:27.039 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:50:27.039 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:50:27.039 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:50:27.039 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:50:27.039 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:50:27.039 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:50:27.039 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:50:27.039 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:27.039 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:27.039 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:50:27.039 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:50:27.040 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:50:27.040 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:50:27.040 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:27.040 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:27.040 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:50:27.040 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:50:27.040 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:50:27.040 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:50:27.040 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:50:27.040 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:50:27.040 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:50:27.040 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:50:27.040 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:27.040 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:27.040 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/ 2024-01-13T09:50:27.041 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: lvm/zap.py", line 302, in zap_osd 2024-01-13T09:50:27.041 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:50:27.041 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:50:27.041 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:50:27.041 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:50:27.041 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:50:27.041 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:50:27.041 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: main() 2024-01-13T09:50:27.041 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:50:27.041 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: r = ctx.func(ctx) 2024-01-13T09:50:27.041 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:50:27.041 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:50:27.041 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:50:27.041 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:50:27.042 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:50:27.042 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:50:27.042 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:50:27.042 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:50:27.042 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:50:27.042 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:50:27.042 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:50:27.042 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:50:27.042 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4t8rzoeb:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpn_kkvg16:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:50:27.042 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:50:27.042 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T09:50:27.042 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: if self._apply_service(spec): 2024-01-13T09:50:27.042 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T09:50:27.042 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T09:50:27.043 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T09:50:27.043 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T09:50:27.043 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T09:50:27.043 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T09:50:27.043 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T09:50:27.043 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T09:50:27.043 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T09:50:27.043 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: raise self._value 2024-01-13T09:50:27.043 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T09:50:27.043 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: result = (True, func(*args, **kwds)) 2024-01-13T09:50:27.043 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T09:50:27.043 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: return list(map(*args)) 2024-01-13T09:50:27.043 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:50:27.043 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: return f(*arg) 2024-01-13T09:50:27.043 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:50:27.044 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:50:27.044 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:50:27.044 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: code, '\n'.join(err))) 2024-01-13T09:50:27.044 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:50:27.044 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4t8rzoeb:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpn_kkvg16:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:50:27.044 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:50:27.044 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:50:27.044 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:50:27.044 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 82bb432f-24c0-40d1-8173-c57cd5b133ef 2024-01-13T09:50:27.044 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:50:27.044 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:50:27.044 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:50:27.044 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:50:27.044 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:50:27.045 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr self.prepare() 2024-01-13T09:50:27.045 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:27.045 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:27.045 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:50:27.045 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:50:27.045 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:50:27.045 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:50:27.045 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:50:27.045 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:50:27.045 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:50:27.045 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:50:27.045 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:50:27.045 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:50:27.045 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:50:27.046 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:50:27.046 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:50:27.046 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:50:27.046 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:50:27.046 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:50:27.046 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:50:27.046 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:50:27.046 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:50:27.046 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:50:27.046 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:50:27.046 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:50:27.046 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:50:27.046 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:50:27.046 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:50:27.046 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:50:27.047 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:27.047 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:27.047 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:50:27.047 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:50:27.047 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:50:27.047 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:50:27.047 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:27.047 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:27.047 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:50:27.047 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:50:27.047 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:50:27.047 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:50:27.047 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:50:27.047 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:50:27.047 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:50:27.047 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:50:27.048 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:50:27.048 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:50:27.048 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:50:27.048 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:50:27.048 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:50:27.048 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:50:27.048 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:50:27.048 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:50:27.048 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:50:27.048 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: main() 2024-01-13T09:50:27.048 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:50:27.048 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: r = ctx.func(ctx) 2024-01-13T09:50:27.048 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:50:27.048 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:50:27.048 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:50:27.049 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:50:27.049 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:50:27.049 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:50:27.049 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:50:27.049 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:50:27.049 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:50:27.049 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:50:27.049 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:50:27.049 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:50:27.049 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4t8rzoeb:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpn_kkvg16:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:50:27.049 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:26 smithi084 conmon[33408]: cluster 2024-01-13T09:50:25.276711+0000 mgr.smithi084.itrhok (mgr.14180) 162 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:27.277 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:50:28.866 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:28 smithi084 conmon[33408]: cluster 2024-01-13T09:50:27.277231+0000 mgr.smithi084.itrhok (mgr.14180) 163 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:28.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:28 smithi102 conmon[37544]: cluster 2024-01-13T09:50:27.277231+0000 mgr.smithi084.itrhok (mgr.14180) 163 : cluster [DBG] 2024-01-13T09:50:28.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:28 smithi102 conmon[37544]: pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:29.193 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:50:29.925 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:29 smithi084 conmon[33408]: audit 2024-01-13T09:50:29.202041+0000 mon.smithi084 (mon.0) 482 : audit [DBG] from='client.? 172.21.15.84:0/2307208542' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:50:29.939 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:50:29.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:29 smithi102 conmon[37544]: audit 2024-01-13T09:50:29.202041+0000 mon.smithi084 (mon.0) 482 2024-01-13T09:50:29.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:29 smithi102 conmon[37544]: : audit [DBG] from='client.? 172.21.15.84:0/2307208542' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:50:30.940 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:50:30.972 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:30 smithi084 conmon[33408]: cluster 2024-01-13T09:50:29.277617+0000 mgr.smithi084.itrhok (mgr.14180) 164 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:30.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:30 smithi102 conmon[37544]: cluster 2024-01-13T09:50:29.277617+0000 mgr.smithi084.itrhok (mgr.14180) 164 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:30.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:30 smithi102 conmon[37544]: 2024-01-13T09:50:32.794 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:50:32.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:32 smithi102 conmon[37544]: cluster 2024-01-13T09:50:31.278014+0000 mgr.smithi084.itrhok (mgr.14180) 165 : cluster 2024-01-13T09:50:32.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:32 smithi102 conmon[37544]: [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:33.029 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:32 smithi084 conmon[33408]: cluster 2024-01-13T09:50:31.278014+0000 mgr.smithi084.itrhok (mgr.14180) 165 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:33.490 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:50:33.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:33 smithi084 conmon[33408]: audit 2024-01-13T09:50:32.802384+0000 mon.smithi084 (mon.0) 483 : audit [DBG] from='client.? 172.21.15.84:0/2798228444' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:50:33.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:33 smithi102 conmon[37544]: audit 2024-01-13T09:50:32.802384+0000 mon.smithi084 (mon.0) 483 : audit [DBG] from='client.? 172.21.15.84:0/2798228444' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:50:34.491 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:50:34.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:34 smithi084 conmon[33408]: cluster 2024-01-13T09:50:33.278488+0000 mgr.smithi084.itrhok (mgr.14180) 166 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:34.985 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:34 smithi102 conmon[37544]: cluster 2024-01-13T09:50:33.278488+0000 mgr.smithi084.itrhok (mgr.14180) 166 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:36.480 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:50:36.485 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:36 smithi102 conmon[37544]: cluster 2024-01-13T09:50:35.278922+0000 mgr.smithi084.itrhok (mgr.14180) 167 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:36.510 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:36 smithi084 conmon[33408]: cluster 2024-01-13T09:50:35.278922+0000 mgr.smithi084.itrhok (mgr.14180) 167 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:37.088 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:50:37.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:37 smithi084 conmon[33408]: audit 2024-01-13T09:50:36.488580+0000 mon.smithi084 (mon.0) 484 : audit 2024-01-13T09:50:37.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:37 smithi084 conmon[33408]: [DBG] from='client.? 172.21.15.84:0/2099372164' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:50:37.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:37 smithi102 conmon[37544]: audit 2024-01-13T09:50:36.488580+0000 mon.smithi084 (mon.0) 484 : audit [DBG] from='client.? 172.21.15.84:0/2099372164' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:50:38.088 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:50:38.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:38 smithi084 conmon[33408]: cluster 2024-01-13T09:50:37.279299+0000 mgr.smithi084.itrhok (mgr.14180) 168 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:38.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:38 smithi102 conmon[37544]: cluster 2024-01-13T09:50:37.279299+0000 mgr.smithi084.itrhok (mgr.14180) 168 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:40.053 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:50:40.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:40 smithi084 conmon[33408]: cluster 2024-01-13T09:50:39.279902+0000 mgr.smithi084.itrhok (mgr.14180) 169 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:40.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:40 smithi084 conmon[33408]: audit 2024-01-13T09:50:40.061400+0000 mon.smithi084 (mon.0) 485 : audit [DBG] from='client.? 172.21.15.84:0/929470771' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:50:40.671 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:50:40.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:40 smithi102 conmon[37544]: cluster 2024-01-13T09:50:39.279902+0000 mgr.smithi084.itrhok (mgr.14180) 169 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:40.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:40 smithi102 conmon[37544]: audit 2024-01-13T09:50:40.061400+0000 mon.smithi084 (mon.0) 485 : audit [DBG] from='client.? 172.21.15.84:0/929470771' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:50:41.673 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:50:42.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:42 smithi084 conmon[33408]: cluster 2024-01-13T09:50:41.280351+0000 mgr.smithi084.itrhok (mgr 2024-01-13T09:50:42.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:42 smithi084 conmon[33408]: .14180) 170 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:42.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:42 smithi102 conmon[37544]: cluster 2024-01-13T09:50:41.280351+0000 mgr.smithi084.itrhok (mgr.14180) 170 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:43.618 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:50:44.236 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:50:44.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:44 smithi084 conmon[33408]: cluster 2024-01-13T09:50:43.280710+0000 mgr.smithi084.itrhok (mgr.14180) 171 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:44.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:44 smithi084 conmon[33408]: audit 2024-01-13T09:50:43.626113+0000 mon.smithi084 (mon.0) 486 : audit [DBG] from='client.? 172.21.15.84:0/1300642344' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:50:44.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:44 smithi102 conmon[37544]: cluster 2024-01-13T09:50:43.280710+0000 mgr.smithi084.itrhok (mgr.14180) 171 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:44.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:44 smithi102 conmon[37544]: audit 2024-01-13T09:50:43.626113+0000 mon.smithi084 (mon.0) 486 : audit [DBG] from='client.? 172.21.15.84:0/1300642344' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:50:45.237 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:50:46.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:46 smithi102 conmon[37544]: cluster 2024-01-13T09:50:45.281122+0000 mgr.smithi084.itrhok (mgr.14180) 172 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:46.528 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:46 smithi084 conmon[33408]: cluster 2024-01-13T09:50:45.281122+0000 mgr.smithi084.itrhok (mgr.14180) 172 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:47.271 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:50:47.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:47 smithi102 conmon[37544]: audit 2024-01-13T09:50:47.278558+0000 mon.smithi084 (mon.0) 487 : audit [DBG] from='client.? 172.21.15.84:0/4201502501' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:50:47.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:47 smithi084 conmon[33408]: audit 2024-01-13T09:50:47.278558+0000 mon.smithi084 (mon.0) 487 : audit [DBG] from='client.? 172.21.15.84:0/4201502501' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:50:47.984 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:50:48.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:48 smithi102 conmon[37544]: cluster 2024-01-13T09:50:47.281468+0000 mgr.smithi084.itrhok (mgr.14180) 173 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:48.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:48 smithi084 conmon[33408]: cluster 2024-01-13T09:50:47.281468+0000 mgr.smithi084.itrhok (mgr.14180) 173 2024-01-13T09:50:48.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:48 smithi084 conmon[33408]: : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:48.985 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:50:50.474 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:50 smithi084 conmon[33408]: cluster 2024-01-13T09:50:49.281816+0000 mgr.smithi084.itrhok (mgr.14180) 174 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:50.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:50 smithi102 conmon[37544]: cluster 2024-01-13T09:50:49.281816+0000 mgr.smithi084.itrhok (mgr.14180) 174 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:50.829 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:50:51.515 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:50:51.735 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:51 smithi102 conmon[37544]: audit 2024-01-13T09:50:50.837787+0000 mon.smithi084 (mon.0) 488 : audit [DBG] from='client.? 172.21.15.84:0/2946778265' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:50:51.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:51 smithi084 conmon[33408]: audit 2024-01-13T09:50:50.837787+0000 mon.smithi084 (mon.0) 488 : 2024-01-13T09:50:51.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:51 smithi084 conmon[33408]: audit [DBG] from='client.? 172.21.15.84:0/2946778265' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:50:52.516 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:50:52.548 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:52 smithi084 conmon[33408]: cluster 2024-01-13T09:50:51.282427+0000 mgr.smithi084.itrhok (mgr.14180) 175 : cluster 2024-01-13T09:50:52.548 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:52 smithi084 conmon[33408]: [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:52.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:52 smithi102 conmon[37544]: cluster 2024-01-13T09:50:51.282427+0000 mgr.smithi084.itrhok (mgr.14180) 175 : 2024-01-13T09:50:52.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:52 smithi102 conmon[37544]: cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:54.440 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:50:54.465 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:54 smithi084 conmon[33408]: cluster 2024-01-13T09:50:53.282816+0000 mgr.smithi084.itrhok (mgr.14180) 176 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:54.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:54 smithi102 conmon[37544]: cluster 2024-01-13T09:50:53.282816+0000 mgr.smithi084.itrhok (mgr.14180) 176 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:54.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:54 smithi102 conmon[37544]: 2024-01-13T09:50:55.118 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:50:55.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:55 smithi084 conmon[33408]: audit 2024-01-13T09:50:54.447770+0000 mon.smithi084 (mon.0) 489 : audit [DBG] from='client.? 172.21.15.84:0/1933117088' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:50:55.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:55 smithi102 conmon[37544]: audit 2024-01-13T09:50:54.447770+0000 mon.smithi084 (mon.0) 489 : 2024-01-13T09:50:55.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:55 smithi102 conmon[37544]: audit [DBG] from='client.? 172.21.15.84:0/1933117088' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:50:56.119 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:50:56.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:56 smithi102 conmon[37544]: cluster 2024-01-13T09:50:55.283251+0000 mgr.smithi084.itrhok (mgr.14180) 177 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:56.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:56 smithi084 conmon[33408]: cluster 2024-01-13T09:50:55.283251+0000 mgr.smithi084.itrhok (mgr.14180) 177 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:58.032 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:50:58.643 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:58 smithi084 conmon[33408]: cluster 2024-01-13T09:50:57.283645+0000 mgr.smithi084.itrhok (mgr.14180) 178 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:58.644 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:50:58 smithi084 conmon[33408]: audit 2024-01-13T09:50:58.040527+0000 mon.smithi084 (mon.0) 490 : audit [DBG] from='client.? 172.21.15.84:0/654884204' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:50:58.660 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:50:58.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:58 smithi102 conmon[37544]: cluster 2024-01-13T09:50:57.283645+0000 mgr.smithi084.itrhok (mgr.14180) 178 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:50:58.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:50:58 smithi102 conmon[37544]: audit 2024-01-13T09:50:58.040527+0000 mon.smithi084 (mon.0) 490 : audit [DBG] from='client.? 172.21.15.84:0/654884204' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:50:59.663 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:51:00.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:00 smithi084 conmon[33408]: cluster 2024-01-13T09:50:59.284035+0000 mgr.smithi084.itrhok (mgr.14180 2024-01-13T09:51:00.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:00 smithi084 conmon[33408]: ) 179 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:00.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:00 smithi102 conmon[37544]: cluster 2024-01-13T09:50:59.284035+0000 mgr.smithi084.itrhok (mgr.14180) 179 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:01.610 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:51:02.216 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:51:02.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:02 smithi084 conmon[33408]: cluster 2024-01-13T09:51:01.284587+0000 mgr.smithi084.itrhok (mgr.14180) 180 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:02.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:02 smithi084 conmon[33408]: audit 2024-01-13T09:51:01.617733+0000 mon.smithi084 (mon.0) 491 : audit [DBG] from='client.? 172.21.15.84:0/519073179' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:51:02.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:02 smithi102 conmon[37544]: cluster 2024-01-13T09:51:01.284587+0000 mgr.smithi084.itrhok (mgr.14180) 180 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:02.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:02 smithi102 conmon[37544]: audit 2024-01-13T09:51:01.617733+0000 mon.smithi084 (mon.0) 491 : audit [DBG] from='client.? 172.21.15.84:0/519073179' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:51:03.218 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:51:04.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:04 smithi084 conmon[33408]: cluster 2024-01-13T09:51:03.285120+0000 mgr.smithi084.itrhok (mgr 2024-01-13T09:51:04.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:04 smithi084 conmon[33408]: .14180) 181 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:04.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:04 smithi102 conmon[37544]: cluster 2024-01-13T09:51:03.285120+0000 mgr.smithi084.itrhok (mgr.14180) 181 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:05.189 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:51:05.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:05 smithi084 conmon[33408]: audit 2024-01-13T09:51:05.197257+0000 mon.smithi084 (mon.0) 492 : audit [DBG] from='client.? 172.21.15.84:0/1393963679' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:51:05.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:05 smithi102 conmon[37544]: audit 2024-01-13T09:51:05.197257+0000 mon.smithi084 (mon.0) 492 : audit [DBG] from='client.? 172.21.15.84:0/1393963679' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:51:05.800 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:51:06.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:06 smithi102 conmon[37544]: cluster 2024-01-13T09:51:05.285564+0000 mgr.smithi084.itrhok (mgr.14180) 182 : cluster 2024-01-13T09:51:06.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:06 smithi102 conmon[37544]: [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:06.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:06 smithi084 conmon[33408]: cluster 2024-01-13T09:51:05.285564+0000 mgr.smithi084.itrhok (mgr.14180) 182 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:06.802 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:51:08.335 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:08 smithi084 conmon[33408]: cluster 2024-01-13T09:51:07.285924+0000 mgr.smithi084.itrhok (mgr.14180) 183 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:08.335 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:08 smithi084 conmon[33408]: 2024-01-13T09:51:08.728 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:51:08.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:08 smithi102 conmon[37544]: cluster 2024-01-13T09:51:07.285924+0000 mgr.smithi084.itrhok (mgr.14180) 183 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:09.345 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:51:09.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:09 smithi102 conmon[37544]: audit 2024-01-13T09:51:08.734982+0000 mon.smithi084 (mon.0) 493 : audit 2024-01-13T09:51:09.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:09 smithi102 conmon[37544]: [DBG] from='client.? 172.21.15.84:0/372646235' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:51:09.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:09 smithi084 conmon[33408]: audit 2024-01-13T09:51:08.734982+0000 mon.smithi084 (mon.0) 493 : audit [DBG] from='client.? 172.21.15.84:0/372646235' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:51:10.346 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:51:10.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:10 smithi102 conmon[37544]: cluster 2024-01-13T09:51:09.286313+0000 mgr.smithi084.itrhok (mgr.14180) 184 : cluster [DBG] 2024-01-13T09:51:10.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:10 smithi102 conmon[37544]: pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:10.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:10 smithi084 conmon[33408]: cluster 2024-01-13T09:51:09.286313+0000 mgr.smithi084.itrhok (mgr.14180) 184 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:12.275 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:51:12.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:12 smithi102 conmon[37544]: cluster 2024-01-13T09:51:11.286835+0000 mgr.smithi084.itrhok (mgr.14180) 185 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:12.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:12 smithi102 conmon[37544]: audit 2024-01-13T09:51:12.283574+0000 mon.smithi084 2024-01-13T09:51:12.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:12 smithi102 conmon[37544]: (mon.0) 494 : audit [DBG] from='client.? 172.21.15.84:0/2793156276' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:51:12.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:12 smithi084 conmon[33408]: cluster 2024-01-13T09:51:11.286835+0000 mgr.smithi084.itrhok (mgr.14180) 185 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:12.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:12 smithi084 conmon[33408]: audit 2024-01-13T09:51:12.283574+0000 mon.smithi084 (mon.0) 494 : audit [DBG] from='client.? 172.21.15.84:0/2793156276' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:51:13.031 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:51:14.033 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:51:14.702 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:14 smithi084 conmon[33408]: cluster 2024-01-13T09:51:13.287313+0000 mgr.smithi084.itrhok (mgr.14180 2024-01-13T09:51:14.702 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:14 smithi084 conmon[33408]: ) 186 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:14.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:14 smithi102 conmon[37544]: cluster 2024-01-13T09:51:13.287313+0000 mgr.smithi084.itrhok (mgr.14180) 186 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:15.916 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:51:16.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:16 smithi102 conmon[37544]: cluster 2024-01-13T09:51:15.288002+0000 mgr.smithi084.itrhok (mgr.14180) 187 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:16.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:16 smithi102 conmon[37544]: audit 2024-01-13T09:51:15.924476+0000 mon.smithi084 (mon.0) 495 : audit [DBG] from='client.? 172.21.15.84:0/3059002848' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:51:16.569 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:16 smithi084 conmon[33408]: cluster 2024-01-13T09:51:15.288002+0000 mgr.smithi084.itrhok (mgr.14180) 187 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:16.569 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:16 smithi084 conmon[33408]: audit 2024-01-13T09:51:15.924476+0000 mon.smithi084 (mon.0) 495 : audit [DBG] from='client.? 172.21.15.84:0/3059002848' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:51:16.585 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:51:17.586 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:51:18.497 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:18 smithi084 conmon[33408]: cluster 2024-01-13T09:51:17.288644+0000 mgr.smithi084.itrhok (mgr.14180 2024-01-13T09:51:18.497 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:18 smithi084 conmon[33408]: ) 188 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:18.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:18 smithi102 conmon[37544]: cluster 2024-01-13T09:51:17.288644+0000 mgr.smithi084.itrhok (mgr.14180) 188 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:19.587 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:51:20.212 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:51:20.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:20 smithi084 conmon[33408]: cluster 2024-01-13T09:51:19.289145+0000 mgr.smithi084.itrhok (mgr.14180) 189 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:20.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:20 smithi084 conmon[33408]: audit 2024-01-13T09:51:19.595126+0000 mon.smithi084 (mon.0) 496 : audit [DBG] from='client.? 172.21.15.84:0/1967525086' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:51:20.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:20 smithi102 conmon[37544]: cluster 2024-01-13T09:51:19.289145+0000 mgr.smithi084.itrhok (mgr.14180) 189 : cluster [DBG] 2024-01-13T09:51:20.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:20 smithi102 conmon[37544]: pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:20.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:20 smithi102 conmon[37544]: audit 2024-01-13T09:51:19.595126+0000 mon.smithi084 (mon.0) 496 : audit [DBG] from='client.? 172.21.15.84:0/1967525086' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:51:21.214 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:51:22.529 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:22 smithi084 conmon[33408]: cluster 2024-01-13T09:51:21.289567+0000 mgr.smithi084.itrhok (mgr.14180) 190 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:22.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:22 smithi102 conmon[37544]: cluster 2024-01-13T09:51:21.289567+0000 mgr.smithi084.itrhok (mgr.14180) 190 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:23.256 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:51:23.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:23 smithi084 conmon[33408]: audit 2024-01-13T09:51:23.263514+0000 mon.smithi084 2024-01-13T09:51:23.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:23 smithi084 conmon[33408]: (mon.0) 497 : audit [DBG] from='client.? 172.21.15.84:0/2443582099' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:51:23.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:23 smithi102 conmon[37544]: audit 2024-01-13T09:51:23.263514+0000 mon.smithi084 (mon.0) 497 : audit [DBG] from='client.? 172.21.15.84:0/2443582099' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:51:23.889 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:51:24.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:24 smithi102 conmon[37544]: cluster 2024-01-13T09:51:23.289994+0000 mgr.smithi084.itrhok (mgr.14180) 191 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:24.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:24 smithi084 conmon[33408]: cluster 2024-01-13T09:51:23.289994+0000 mgr.smithi084.itrhok (mgr.14180) 191 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:24.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:24 smithi084 conmon[33408]: 2024-01-13T09:51:24.891 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:51:25.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:25 smithi102 conmon[37544]: audit 2024-01-13T09:51:25.277999+0000 mon.smithi084 (mon.0) 498 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T09:51:25.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:25 smithi084 conmon[33408]: audit 2024-01-13T09:51:25 2024-01-13T09:51:25.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:25 smithi084 conmon[33408]: .277999+0000 mon.smithi084 (mon.0) 498 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T09:51:26.465 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:26 smithi084 conmon[33408]: cluster 2024-01-13T09:51:25.290354+0000 mgr.smithi084.itrhok (mgr.14180) 192 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:26.465 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:26 smithi084 conmon[33408]: audit 2024-01-13T09:51:25.598285+0000 mon.smithi084 (mon.0) 499 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:51:26.465 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:26 smithi084 conmon[33408]: audit 2024-01-13T09:51:25.599017+0000 mon.smithi084 (mon.0) 500 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T09:51:26.465 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:26 smithi084 conmon[33408]: audit 2024-01-13T09:51:25.605002+0000 mon.smithi084 (mon.0) 501 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:51:26.465 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:26 smithi084 conmon[33408]: cluster 2024-01-13T09:51:25.605307+0000 mgr.smithi084.itrhok (mgr.14180) 193 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:26.465 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:26 smithi084 conmon[33408]: audit 2024-01-13T09:51:25.608288+0000 mon.smithi084 (mon.0) 502 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T09:51:26.465 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:26 smithi084 conmon[33408]: audit 2024-01-13T09:51:25.610867+0000 mon.smithi084 (mon.0) 503 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:51:26.465 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:26 smithi084 conmon[33408]: audit 2024-01-13T09:51:25.611789+0000 mon.smithi084 (mon.0) 504 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:51:26.465 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:26 smithi084 conmon[33408]: audit 2024-01-13T09:51:25.612026+0000 mon.smithi084 (mon.0) 505 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:51:26.465 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:26 smithi084 conmon[33408]: audit 2024-01-13T09:51:25.612467+0000 mon.smithi084 (mon.0) 506 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:51:26.465 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:26 smithi084 conmon[33408]: cluster 2024-01-13T09:51:26.285841+0000 mon.smithi084 (mon.0) 507 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T09:51:26.465 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:26 smithi084 conmon[33408]: cluster 2024-01-13T09:51:26.285868+0000 mon.smithi084 (mon.0) 508 : cluster [INF] Cluster is now healthy 2024-01-13T09:51:26.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:26 smithi102 conmon[37544]: cluster 2024-01-13T09:51:25.290354+0000 mgr.smithi084.itrhok (mgr.14180) 192 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:26.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:26 smithi102 conmon[37544]: audit 2024-01-13T09:51:25.598285+0000 mon.smithi084 (mon.0) 499 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:51:26.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:26 smithi102 conmon[37544]: audit 2024-01-13T09:51:25.599017+0000 mon.smithi084 (mon.0) 500 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T09:51:26.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:26 smithi102 conmon[37544]: audit 2024-01-13T09:51:25.605002+0000 mon.smithi084 (mon.0) 501 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:51:26.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:26 smithi102 conmon[37544]: cluster 2024-01-13T09:51:25.605307+0000 mgr.smithi084.itrhok (mgr.14180) 193 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:26.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:26 smithi102 conmon[37544]: audit 2024-01-13T09:51:25.608288+0000 mon.smithi084 (mon.0) 502 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T09:51:26.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:26 smithi102 conmon[37544]: audit 2024-01-13T09:51:25.610867+0000 mon.smithi084 (mon.0) 503 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:51:26.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:26 smithi102 conmon[37544]: audit 2024-01-13T09:51:25.611789+0000 mon.smithi084 (mon.0) 504 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:51:26.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:26 smithi102 conmon[37544]: audit 2024-01-13T09:51:25.612026+0000 mon.smithi084 (mon.0) 505 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:51:26.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:26 smithi102 conmon[37544]: audit 2024-01-13T09:51:25.612467+0000 mon.smithi084 (mon.0) 506 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:51:26.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:26 smithi102 conmon[37544]: cluster 2024-01-13T09:51:26.285841+0000 mon.smithi084 (mon.0) 507 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T09:51:26.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:26 smithi102 conmon[37544]: cluster 2024-01-13T09:51:26.285868+0000 mon.smithi084 (mon.0) 508 : cluster [INF] Cluster is now healthy 2024-01-13T09:51:26.809 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:51:27.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:27 smithi102 conmon[37544]: audit 2024-01-13T09:51:26.816559+0000 mon.smithi084 (mon.0) 509 : audit [DBG] from='client.? 172.21.15.84:0/2058038794' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:51:27.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:27 smithi102 conmon[37544]: audit 2024-01-13T09:51:27.208078+0000 mon.smithi084 (mon.0) 510 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:51:27.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:27 smithi084 conmon[33408]: audit 2024-01-13T09:51:26.816559+0000 mon.smithi084 (mon.0) 509 : audit [DBG] from='client.? 172.21.15.84:0/2058038794' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:51:27.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:27 smithi084 conmon[33408]: audit 2024-01-13T09:51:27.208078+0000 mon.smithi084 (mon.0) 510 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:51:27.952 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:51:28.659 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:28 smithi102 conmon[37544]: cluster 2024-01-13T09:51:27.605579+0000 mgr.smithi084.itrhok (mgr.14180) 194 : cluster 2024-01-13T09:51:28.843 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:28 smithi102 conmon[37544]: [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:28.843 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:28 smithi102 conmon[37544]: audit 2024-01-13T09:51:28.082228+0000 mon.smithi102 (mon.1) 10 : audit [INF] from='client.? 172.21.15.102:0/3152884479' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2357a1f6-974e-4dde-98be-3405a22a8a92"}]: dispatch 2024-01-13T09:51:28.843 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:28 smithi102 conmon[37544]: audit 2024-01-13T09:51:28.082580+0000 mon.smithi084 ( 2024-01-13T09:51:28.843 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:28 smithi102 conmon[37544]: mon.0) 511 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2357a1f6-974e-4dde-98be-3405a22a8a92"}]: dispatch 2024-01-13T09:51:28.843 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:28 smithi102 conmon[37544]: audit 2024-01-13T09:51:28.088699+0000 mon.smithi084 (mon. 2024-01-13T09:51:28.843 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:28 smithi102 conmon[37544]: 0) 512 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2357a1f6-974e-4dde-98be-3405a22a8a92"}]': finished 2024-01-13T09:51:28.843 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:28 smithi102 conmon[37544]: cluster 2024-01-13T 2024-01-13T09:51:28.843 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:28 smithi102 conmon[37544]: 09:51:28.088748+0000 mon.smithi084 (mon.0) 513 : cluster [DBG] osdmap e22: 1 total, 0 up, 1 in 2024-01-13T09:51:28.843 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:28 smithi102 conmon[37544]: audit 2024-01-13T09:51: 2024-01-13T09:51:28.843 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:28 smithi102 conmon[37544]: 28.088840+0000 mon.smithi084 (mon.0) 514 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:51:28.844 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:28 smithi102 conmon[37544]: 2024-01-13T09:51:28.844 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:28 smithi084 conmon[33408]: cluster 2024-01-13T09:51:27.605579+0000 mgr.smithi084.itrhok (mgr.14180) 194 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:28.844 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:28 smithi084 conmon[33408]: audit 2024-01-13T09:51:28.082228+0000 mon.smithi102 (mon.1) 10 : audit [INF] from='client.? 172.21.15.102:0/3152884479' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2357a1f6-974e-4dde-98be-3405a22a8a92"}]: dispatch 2024-01-13T09:51:28.845 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:28 smithi084 conmon[33408]: audit 2024-01-13T09:51:28.082580+0000 mon.smithi084 (mon.0) 511 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2357a1f6-974e-4dde-98be-3405a22a8a92"}]: dispatch 2024-01-13T09:51:28.845 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:28 smithi084 conmon[33408]: audit 2024-01-13T09:51:28.088699+0000 mon.smithi084 (mon.0) 512 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2357a1f6-974e-4dde-98be-3405a22a8a92"}]': finished 2024-01-13T09:51:28.845 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:28 smithi084 conmon[33408]: cluster 2024-01-13T09:51:28.088748+0000 mon.smithi084 (mon.0) 513 : cluster [DBG] osdmap e22: 1 total, 0 up, 1 in 2024-01-13T09:51:28.845 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:28 smithi084 conmon[33408]: audit 2024-01-13T09:51:28.088840+0000 mon.smithi084 (mon.0) 514 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:51:28.953 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:51:29.630 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:29 smithi084 conmon[33408]: audit 2024-01-13T09:51:28.374712+0000 mon.smithi102 (mon. 2024-01-13T09:51:29.630 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:29 smithi084 conmon[33408]: 1) 11 : audit [INF] from='client.? 172.21.15.102:0/503238754' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:51:29.630 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:29 smithi084 conmon[33408]: audit 2024-01-13T09:51:28.374801+0000 mon.smithi084 (mon.0) 515 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:51:29.630 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:29 smithi084 conmon[33408]: audit 2024-01-13T09:51:28.383515+0000 mon.smithi084 (mon.0) 516 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:51:29.630 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:29 smithi084 conmon[33408]: cluster 2024-01-13T09:51:28.383584+0000 mon.smithi084 (mon.0) 517 : cluster [DBG] osdmap e23: 0 total, 0 up, 0 in 2024-01-13T09:51:29.630 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:29 smithi084 conmon[33408]: audit 2024-01-13T09:51:29.197580+0000 mon.smithi084 (mon.0) 518 : audit [INF] from='client.? 172.21.15.84:0/1435896079' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "883c1246-140f-4c6d-8a46-6824956519ff"}]: dispatch 2024-01-13T09:51:29.630 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:29 smithi084 conmon[33408]: audit 2024-01-13T09:51:29.201155+0000 mon.smithi084 (mon.0) 519 : audit [INF] from='client.? 172.21.15.84:0/1435896079' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "883c1246-140f-4c6d-8a46-6824956519ff"}]': finished 2024-01-13T09:51:29.630 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:29 smithi084 conmon[33408]: cluster 2024-01-13T09:51:29.201199+0000 mon.smithi084 (mon.0) 520 : cluster [DBG] osdmap e24: 1 total, 0 up, 1 in 2024-01-13T09:51:29.630 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:29 smithi084 conmon[33408]: audit 2024-01-13T09:51:29.201306+0000 mon.smithi084 (mon.0) 521 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:51:29.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:29 smithi102 conmon[37544]: audit 2024-01-13T09:51:28.374712+0000 mon.smithi102 (mon.1) 11 : audit [INF] from='client.? 172.21.15.102:0/503238754' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:51:29.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:29 smithi102 conmon[37544]: audit 2024-01-13T09:51:28.374801+0000 mon.smithi084 2024-01-13T09:51:29.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:29 smithi102 conmon[37544]: (mon.0) 515 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:51:29.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:29 smithi102 conmon[37544]: audit 2024-01-13T09:51:28.383515+0000 mon.smithi084 (mon.0) 516 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:51:29.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:29 smithi102 conmon[37544]: cluster 2024-01-13T09:51:28.383584+0000 mon.smithi084 (mon.0) 517 : cluster [DBG] osdmap e23: 0 total, 0 up, 0 in 2024-01-13T09:51:29.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:29 smithi102 conmon[37544]: audit 2024-01-13T09:51:29.197580+0000 mon.smithi084 (mon.0) 518 : audit [INF] from='client.? 172.21.15.84:0/1435896079' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "883c1246-140f-4c6d-8a46-6824956519ff"}]: dispatch 2024-01-13T09:51:29.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:29 smithi102 conmon[37544]: audit 2024-01-13T09:51:29.201155+0000 mon.smithi084 (mon.0) 519 : audit [INF] from='client.? 172.21.15.84:0/1435896079' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "883c1246-140f-4c6d-8a46-6824956519ff"}]': finished 2024-01-13T09:51:29.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:29 smithi102 conmon[37544]: cluster 2024-01-13T09:51:29.201199+0000 mon.smithi084 (mon.0) 520 : cluster [DBG] osdmap e24: 1 total, 0 up, 1 in 2024-01-13T09:51:29.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:29 smithi102 conmon[37544]: audit 2024-01-13T09:51:29.201306+0000 mon.smithi084 (mon.0) 521 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:51:30.483 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: cephadm 2024-01-13T09:51:29.038709+0000 mgr.smithi084.itrhok (mgr.14180) 195 : cephadm [ERR] executing create_from_spec_one(([('smithi084', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi102', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:51:30.483 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:51:30.483 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:51:30.483 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: return f(*arg) 2024-01-13T09:51:30.483 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:51:30.483 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:51:30.483 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:51:30.483 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: code, '\n'.join(err))) 2024-01-13T09:51:30.483 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:51:30.484 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpwv1eypyr:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpg57g68yx:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:51:30.484 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:51:30.484 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:51:30.484 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:51:30.484 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 2357a1f6-974e-4dde-98be-3405a22a8a92 2024-01-13T09:51:30.484 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:51:30.484 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:51:30.484 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:51:30.484 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:51:30.484 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:51:30.484 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr self.prepare() 2024-01-13T09:51:30.484 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:51:30.485 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:51:30.485 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:51:30.485 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:51:30.485 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:51:30.485 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:51:30.485 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:51:30.485 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:51:30.485 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:51:30.485 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:51:30.485 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:51:30.485 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:51:30.485 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:51:30.485 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:51:30.485 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:51:30.485 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:51:30.486 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:51:30.486 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:51:30.486 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:51:30.486 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:51:30.486 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:51:30.486 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:51:30.486 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:51:30.486 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:51:30.486 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:51:30.486 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:51:30.486 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:51:30.486 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:51:30.486 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:51:30.486 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:51:30.487 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:51:30.487 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:51:30.487 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:51:30.487 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:51:30.487 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:51:30.487 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:51:30.487 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:51:30.487 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:51:30.488 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:51:30.488 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:51:30.488 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:51:30.488 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:51:30.488 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:51:30.488 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:51:30.488 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:51:30.488 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:51:30.488 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:51:30.488 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:51:30.488 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:51:30.488 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:51:30.488 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:51:30.488 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:51:30.489 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: main() 2024-01-13T09:51:30.489 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:51:30.489 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: r = ctx.func(ctx) 2024-01-13T09:51:30.489 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:51:30.489 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:51:30.489 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:51:30.489 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:51:30.489 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:51:30.489 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:51:30.489 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-8777 2024-01-13T09:51:30.489 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: 4f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:51:30.489 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:51:30.489 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:51:30.489 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:51:30.490 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:51:30.490 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:51:30.490 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpwv1eypyr:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpg57g68yx:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:51:30.490 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: audit 2024-01-13T09:51:29.503854+0000 mon.smithi084 (mon.0) 522 : audit [INF] from='client.? 172.21.15.84:0/194949865' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:51:30.490 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: audit 2024-01-13T09:51:29.507950+0000 mon.smithi084 (mon.0) 523 : audit [INF] from='client.? 172.21.15.84:0/194949865' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:51:30.490 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: cluster 2024-01-13T09:51:29.507997+0000 mon.smithi084 (mon.0) 524 : cluster [DBG] osdmap e25: 0 total, 0 up, 0 in 2024-01-13T09:51:30.490 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:30 smithi084 conmon[33408]: cluster 2024-01-13T09:51:29.605808+0000 mgr.smithi084.itrhok (mgr.14180) 196 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:30.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: cephadm 2024-01-13T09:51:29.038709+0000 mgr.smithi084.itrhok (mgr.14180) 195 : cephadm [ERR] executing create_from_spec_one(([('smithi084', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi102', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:51:30.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:51:30.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:51:30.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: return f(*arg) 2024-01-13T09:51:30.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:51:30.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:51:30.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:51:30.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: code, '\n'.join(err))) 2024-01-13T09:51:30.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:51:30.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpwv1eypyr:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpg57g68yx:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:51:30.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:51:30.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:51:30.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:51:30.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 2357a1f6-974e-4dde-98be-3405a22a8a92 2024-01-13T09:51:30.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:51:30.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:51:30.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:51:30.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:51:30.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:51:30.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr self.prepare() 2024-01-13T09:51:30.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:51:30.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:51:30.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:51:30.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:51:30.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:51:30.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:51:30.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:51:30.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:51:30.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:51:30.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:51:30.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:51:30.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:51:30.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:51:30.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:51:30.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:51:30.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:51:30.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:51:30.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:51:30.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:51:30.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:51:30.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:51:30.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:51:30.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:51:30.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:51:30.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:51:30.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:51:30.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:51:30.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:51:30.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:51:30.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:51:30.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:51:30.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:51:30.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:51:30.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:51:30.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:51:30.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:51:30.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:51:30.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:51:30.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:51:30.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:51:30.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:51:30.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:51:30.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:51:30.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:51:30.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:51:30.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:51:30.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:51:30.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:51:30.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:51:30.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:51:30.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:51:30.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:51:30.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: main() 2024-01-13T09:51:30.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:51:30.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: r = ctx.func(ctx) 2024-01-13T09:51:30.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:51:30.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:51:30.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:51:30.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:51:30.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:51:30.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:51:30.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-8777 2024-01-13T09:51:30.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: 4f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:51:30.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:51:30.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:51:30.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:51:30.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:51:30.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:51:30.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpwv1eypyr:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpg57g68yx:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:51:30.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: audit 2024-01-13T09:51:29.503854+0000 mon.smithi084 (mon.0) 522 : audit [INF] from='client.? 172.21.15.84:0/194949865' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:51:30.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: audit 2024-01-13T09:51:29.507950+0000 mon.smithi084 (mon.0) 523 : audit [INF] from='client.? 172.21.15.84:0/194949865' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:51:30.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: cluster 2024-01-13T09:51:29.507997+0000 mon.smithi084 (mon.0) 524 : cluster [DBG] osdmap e25: 0 total, 0 up, 0 in 2024-01-13T09:51:30.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:30 smithi102 conmon[37544]: cluster 2024-01-13T09:51:29.605808+0000 mgr.smithi084.itrhok (mgr.14180) 196 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:31.306 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:51:31.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: cephadm 2024-01-13T09:51:30.715298+0000 2024-01-13T09:51:31.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: mgr.smithi084.itrhok (mgr.14180) 197 : cephadm [ERR] executing create_from_spec_one(([('smithi084', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi102', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:51:31.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:51:31.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:51:31.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: return f(*arg) 2024-01-13T09:51:31.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:51:31.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:51:31.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:51:31.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: code, '\n'.join(err))) 2024-01-13T09:51:31.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi084/config 2024-01-13T09:51:31.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp9pg0zels:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp_0jqcmnk:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:51:31.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:51:31.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:51:31.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:51:31.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 883c1246-140f-4c6d-8a46-6824956519ff 2024-01-13T09:51:31.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:51:31.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:51:31.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:51:31.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:51:31.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:51:31.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr self.prepare() 2024-01-13T09:51:31.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:51:31.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:51:31.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:51:31.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:51:31.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:51:31.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:51:31.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:51:31.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:51:31.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:51:31.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:51:31.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:51:31.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:51:31.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:51:31.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:51:31.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:51:31.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:51:31.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:51:31.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:51:31.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:51:31.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:51:31.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:51:31.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:51:31.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:51:31.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:51:31.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:51:31.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:51:31.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:51:31.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:51:31.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:51:31.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:51:31.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:51:31.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:51:31.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:51:31.785 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:51:31.785 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:51:31.785 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:51:31.785 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:51:31.785 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:51:31.785 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:51:31.785 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:51:31.785 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:51:31.785 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:51:31.785 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:51:31.785 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:51:31.785 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:51:31.785 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:51:31.785 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:51:31.785 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:51:31.785 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:51:31.786 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:51:31.786 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:51:31.786 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:51:31.786 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: main() 2024-01-13T09:51:31.786 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:51:31.786 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: r = ctx.func(ctx) 2024-01-13T09:51:31.786 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:51:31.786 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:51:31.786 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:51:31.786 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:51:31.786 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:51:31.786 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:51:31.786 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517 2024-01-13T09:51:31.786 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: 833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:51:31.786 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:51:31.787 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:51:31.787 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:51:31.787 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:51:31.787 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:51:31.787 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp9pg0zels:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp_0jqcmnk:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:51:31.787 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: cephadm 2024-01-13T09:51:30.716637+0000 mgr.smithi084.itrhok (mgr.14180) 198 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T09:51:31.787 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: service_id: all-available-devices 2024-01-13T09:51:31.787 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: service_name: osd.all-available-devices 2024-01-13T09:51:31.787 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: placement: 2024-01-13T09:51:31.787 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: host_pattern: '*' 2024-01-13T09:51:31.787 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: spec: 2024-01-13T09:51:31.787 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: data_devices: 2024-01-13T09:51:31.787 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: all: true 2024-01-13T09:51:31.787 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: filter_logic: AND 2024-01-13T09:51:31.788 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: objectstore: bluestore 2024-01-13T09:51:31.788 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:51:31.788 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpwv1eypyr:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpg57g68yx:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:51:31.788 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:51:31.788 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:51:31.788 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:51:31.788 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 2357a1f6-974e-4dde-98be-3405a22a8a92 2024-01-13T09:51:31.788 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:51:31.788 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:51:31.788 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:51:31.788 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:51:31.788 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:51:31.788 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr self.prepare() 2024-01-13T09:51:31.788 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:51:31.789 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:51:31.789 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:51:31.789 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:51:31.789 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:51:31.789 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:51:31.789 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:51:31.789 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:51:31.789 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:51:31.789 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:51:31.789 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:51:31.789 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:51:31.789 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:51:31.789 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:51:31.789 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:51:31.789 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:51:31.789 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:51:31.790 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:51:31.790 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:51:31.790 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:51:31.790 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:51:31.790 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:51:31.790 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:51:31.790 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:51:31.790 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:51:31.790 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:51:31.790 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:51:31.790 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:51:31.790 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:51:31.790 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:51:31.790 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:51:31.790 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:51:31.791 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:51:31.791 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:51:31.791 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:51:31.791 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:51:31.791 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:51:31.791 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:51:31.791 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:51:31.791 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:51:31.791 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:51:31.791 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:51:31.791 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:51:31.791 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:51:31.791 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:51:31.791 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:51:31.791 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/p 2024-01-13T09:51:31.792 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: odman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:51:31.792 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:51:31.792 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:51:31.792 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:51:31.792 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:51:31.792 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:51:31.792 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: main() 2024-01-13T09:51:31.792 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:51:31.792 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: r = ctx.func(ctx) 2024-01-13T09:51:31.792 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:51:31.792 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:51:31.792 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:51:31.792 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:51:31.793 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:51:31.793 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:51:31.793 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:51:31.793 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:51:31.793 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:51:31.793 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:51:31.793 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:51:31.793 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:51:31.793 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpwv1eypyr:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpg57g68yx:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:51:31.793 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:51:31.793 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T09:51:31.793 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: if self._apply_service(spec): 2024-01-13T09:51:31.793 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T09:51:31.793 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T09:51:31.794 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T09:51:31.794 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T09:51:31.794 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T09:51:31.794 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T09:51:31.794 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T09:51:31.794 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T09:51:31.794 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T09:51:31.794 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: raise self._value 2024-01-13T09:51:31.794 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T09:51:31.794 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: result = (True, func(*args, **kwds)) 2024-01-13T09:51:31.794 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T09:51:31.795 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: return list(map(*args)) 2024-01-13T09:51:31.795 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:51:31.795 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: return f(*arg) 2024-01-13T09:51:31.795 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:51:31.795 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:51:31.795 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:51:31.795 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: code, '\n'.join(err))) 2024-01-13T09:51:31.795 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:51:31.795 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpwv1eypyr:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpg57g68yx:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:51:31.796 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:51:31.796 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:51:31.796 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:51:31.796 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 2357a1f6-974e-4dde-98be-3405a22a8a92 2024-01-13T09:51:31.796 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:51:31.796 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:51:31.796 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:51:31.796 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:51:31.796 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:51:31.796 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr self.prepare() 2024-01-13T09:51:31.796 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:51:31.796 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:51:31.796 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:51:31.796 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:51:31.797 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:51:31.797 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:51:31.797 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:51:31.797 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:51:31.797 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:51:31.797 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:51:31.797 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:51:31.797 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:51:31.797 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:51:31.797 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:51:31.797 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:51:31.797 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:51:31.797 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:51:31.797 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ 2024-01-13T09:51:31.797 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:51:31.798 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:51:31.798 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:51:31.798 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:51:31.798 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:51:31.798 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:51:31.798 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:51:31.798 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:51:31.798 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:51:31.798 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:51:31.798 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:51:31.798 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:51:31.798 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:51:31.798 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:51:31.798 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:51:31.799 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:51:31.799 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:51:31.799 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:51:31.799 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:51:31.799 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:51:31.799 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:51:31.799 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:51:31.799 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:51:31.799 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:51:31.799 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:51:31.799 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:51:31.799 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:51:31.799 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:51:31.799 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:51:31.799 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:51:31.800 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:51:31.800 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:51:31.800 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:51:31.800 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:51:31.800 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:51:31.800 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: main() 2024-01-13T09:51:31.800 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:51:31.800 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: r = ctx.func(ctx) 2024-01-13T09:51:31.800 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:51:31.800 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:51:31.800 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:51:31.800 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:51:31.800 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:51:31.800 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:51:31.800 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:51:31.801 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:51:31.801 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:51:31.801 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:51:31.801 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:51:31.801 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:51:31.801 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpwv1eypyr:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpg57g68yx:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:51:31.801 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: cluster 2024-01-13T09:51:30.717155+0000 mgr.smithi084.itrhok (mgr.14180) 199 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:31.801 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:31 smithi084 conmon[33408]: audit 2024-01-13T09:51:31.312865+0000 mon.smithi084 (mon.0) 525 : audit [DBG] from='client.? 172.21.15.84:0/2437430938' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:51:31.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: cephadm 2024-01-13T09:51:30.715298+0000 mgr.smithi084.itrhok (mgr.14180) 197 : cephadm [ERR] executing create_from_spec_one(([('smithi084', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi102', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:51:31.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:51:31.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:51:31.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: return f(*arg) 2024-01-13T09:51:31.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:51:31.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:51:31.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:51:31.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: code, '\n'.join(err))) 2024-01-13T09:51:31.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi084/config 2024-01-13T09:51:31.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp9pg0zels:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp_0jqcmnk:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:51:31.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:51:31.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:51:31.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:51:31.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 883c1246-140f-4c6d-8a46-6824956519ff 2024-01-13T09:51:31.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:51:31.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:51:31.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:51:31.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:51:31.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:51:31.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr self.prepare() 2024-01-13T09:51:31.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:51:31.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:51:31.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:51:31.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:51:31.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:51:31.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:51:31.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:51:31.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:51:31.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:51:31.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:51:31.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:51:31.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:51:31.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:51:31.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:51:31.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:51:31.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:51:31.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:51:31.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:51:31.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:51:31.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:51:31.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:51:31.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:51:31.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:51:31.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:51:31.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:51:31.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:51:31.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:51:31.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:51:31.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:51:31.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:51:31.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:51:31.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:51:31.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:51:31.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:51:31.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:51:31.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:51:31.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:51:31.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:51:31.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:51:31.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:51:31.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:51:31.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:51:31.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:51:31.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:51:31.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:51:31.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:51:31.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:51:31.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:51:31.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:51:31.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:51:31.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:51:31.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:51:31.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: main() 2024-01-13T09:51:31.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:51:31.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: r = ctx.func(ctx) 2024-01-13T09:51:31.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:51:31.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:51:31.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:51:31.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:51:31.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:51:31.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:51:31.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-8777 2024-01-13T09:51:31.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: 4f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:51:31.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:51:31.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:51:31.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:51:31.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:51:31.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:51:31.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp9pg0zels:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp_0jqcmnk:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:51:31.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: cephadm 2024-01-13T09:51:30.716637+0000 mgr.smithi084.itrhok (mgr.14180) 198 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T09:51:31.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: service_id: all-available-devices 2024-01-13T09:51:31.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: service_name: osd.all-available-devices 2024-01-13T09:51:31.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: placement: 2024-01-13T09:51:31.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: host_pattern: '*' 2024-01-13T09:51:31.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: spec: 2024-01-13T09:51:31.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: data_devices: 2024-01-13T09:51:31.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: all: true 2024-01-13T09:51:31.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: filter_logic: AND 2024-01-13T09:51:31.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: objectstore: bluestore 2024-01-13T09:51:31.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:51:31.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpwv1eypyr:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpg57g68yx:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:51:31.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:51:31.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:51:31.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:51:31.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 2357a1f6-974e-4dde-98be-3405a22a8a92 2024-01-13T09:51:31.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:51:31.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:51:31.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:51:31.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:51:31.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:51:31.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr self.prepare() 2024-01-13T09:51:31.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:51:31.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:51:31.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:51:31.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:51:31.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:51:31.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:51:31.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:51:31.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:51:31.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:51:31.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:51:31.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:51:31.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:51:31.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:51:31.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:51:31.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:51:31.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:51:31.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:51:31.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:51:31.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:51:31.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:51:31.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:51:31.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:51:31.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:51:31.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:51:31.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:51:31.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:51:31.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:51:32.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:51:32.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:51:32.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:51:32.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:51:32.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:51:32.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:51:32.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:51:32.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:51:32.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:51:32.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:51:32.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:51:32.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:51:32.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:51:32.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:51:32.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:51:32.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:51:32.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:51:32.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:51:32.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/ 2024-01-13T09:51:32.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: lvm/zap.py", line 302, in zap_osd 2024-01-13T09:51:32.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:51:32.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:51:32.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:51:32.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:51:32.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:51:32.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:51:32.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: main() 2024-01-13T09:51:32.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:51:32.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: r = ctx.func(ctx) 2024-01-13T09:51:32.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:51:32.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:51:32.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:51:32.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:51:32.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:51:32.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:51:32.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:51:32.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:51:32.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:51:32.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:51:32.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:51:32.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:51:32.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpwv1eypyr:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpg57g68yx:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:51:32.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:51:32.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T09:51:32.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: if self._apply_service(spec): 2024-01-13T09:51:32.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T09:51:32.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T09:51:32.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T09:51:32.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T09:51:32.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T09:51:32.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T09:51:32.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T09:51:32.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T09:51:32.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T09:51:32.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: raise self._value 2024-01-13T09:51:32.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T09:51:32.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: result = (True, func(*args, **kwds)) 2024-01-13T09:51:32.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T09:51:32.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: return list(map(*args)) 2024-01-13T09:51:32.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:51:32.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: return f(*arg) 2024-01-13T09:51:32.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:51:32.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:51:32.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:51:32.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: code, '\n'.join(err))) 2024-01-13T09:51:32.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:51:32.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpwv1eypyr:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpg57g68yx:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:51:32.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:51:32.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:51:32.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:51:32.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 2357a1f6-974e-4dde-98be-3405a22a8a92 2024-01-13T09:51:32.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:51:32.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:51:32.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:51:32.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:51:32.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:51:32.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr self.prepare() 2024-01-13T09:51:32.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:51:32.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:51:32.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:51:32.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:51:32.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:51:32.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:51:32.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:51:32.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:51:32.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:51:32.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:51:32.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:51:32.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:51:32.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:51:32.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:51:32.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:51:32.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:51:32.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:51:32.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:51:32.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:51:32.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:51:32.009 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:51:32.009 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:51:32.009 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:51:32.009 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:51:32.009 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:51:32.009 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:51:32.009 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:51:32.009 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:51:32.009 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:51:32.009 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:51:32.009 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:51:32.009 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:51:32.010 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:51:32.010 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:51:32.010 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:51:32.010 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:51:32.010 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:51:32.010 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:51:32.010 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:51:32.010 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:51:32.010 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:51:32.010 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:51:32.010 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:51:32.011 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:51:32.011 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:51:32.011 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:51:32.011 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:51:32.011 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:51:32.011 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:51:32.011 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:51:32.011 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:51:32.011 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:51:32.011 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:51:32.011 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: main() 2024-01-13T09:51:32.011 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:51:32.011 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: r = ctx.func(ctx) 2024-01-13T09:51:32.012 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:51:32.012 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:51:32.012 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:51:32.012 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:51:32.012 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:51:32.012 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:51:32.012 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:51:32.012 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:51:32.012 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:51:32.012 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:51:32.012 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:51:32.013 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:51:32.013 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpwv1eypyr:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpg57g68yx:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:51:32.013 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: cluster 2024-01-13T09:51:30.717155+0000 mgr.smithi084.itrhok (mgr.14180) 199 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:32.013 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:31 smithi102 conmon[37544]: audit 2024-01-13T09:51:31.312865+0000 mon.smithi084 (mon.0) 525 : audit [DBG] from='client.? 172.21.15.84:0/2437430938' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:51:32.106 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:51:32.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:32 smithi084 conmon[33408]: cluster 2024-01-13T09:51:31.506615+0000 mon.smithi084 (mon.0) 526 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T09:51:32.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:32 smithi102 conmon[37544]: cluster 2024-01-13T09:51:31.506615+0000 mon.smithi084 (mon.0) 526 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T09:51:33.107 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:51:33.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:33 smithi084 conmon[33408]: cluster 2024-01-13T09:51:32.717556+0000 mgr.smithi084.itrhok (mgr.14180) 200 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:33.986 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:33 smithi102 conmon[37544]: cluster 2024-01-13T09:51:32.717556+0000 mgr.smithi084.itrhok (mgr.14180) 200 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:35.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:34 smithi084 conmon[33408]: cluster 2024-01-13T09:51:34.717935+0000 mgr.smithi084.itrhok (mgr.14180) 201 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:35.087 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:51:35.236 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:34 smithi102 conmon[37544]: cluster 2024-01-13T09:51:34.717935+0000 mgr.smithi084.itrhok (mgr.14180) 201 : cluster [DBG] 2024-01-13T09:51:35.236 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:34 smithi102 conmon[37544]: pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:35.811 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:51:36.112 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:35 smithi102 conmon[37544]: audit 2024-01-13T09:51:35.094083+0000 mon.smithi084 (mon.0) 527 : audit 2024-01-13T09:51:36.112 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:35 smithi102 conmon[37544]: [DBG] from='client.? 172.21.15.84:0/468205498' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:51:36.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:35 smithi084 conmon[33408]: audit 2024-01-13T09:51:35.094083+0000 mon.smithi084 ( 2024-01-13T09:51:36.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:35 smithi084 conmon[33408]: mon.0) 527 : audit [DBG] from='client.? 172.21.15.84:0/468205498' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:51:36.813 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:51:37.236 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:36 smithi102 conmon[37544]: cluster 2024-01-13T09:51:36.718583+0000 mgr.smithi084.itrhok (mgr.14180) 202 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:37.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:36 smithi084 conmon[33408]: cluster 2024-01-13T09:51:36.718583+0000 mgr.smithi084.itrhok (mgr.14180) 202 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:38.778 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:51:39.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:38 smithi102 conmon[37544]: cluster 2024-01-13T09:51:38.719140+0000 mgr.smithi084.itrhok (mgr.14180) 203 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:39.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:38 smithi102 conmon[37544]: audit 2024-01-13T09:51:38.785101+0000 mon.smithi084 (mon.0) 528 : audit [DBG] from='client.? 172.21.15.84:0/207746548' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:51:39.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:38 smithi084 conmon[33408]: cluster 2024-01-13T09:51:38.719140+0000 mgr.smithi084.itrhok 2024-01-13T09:51:39.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:38 smithi084 conmon[33408]: (mgr.14180) 203 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:39.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:38 smithi084 conmon[33408]: audit 2024-01-13T09:51:38.785101+0000 mon.smithi084 (mon.0) 528 : audit [DBG] from='client.? 172.21.15.84:0/207746548' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:51:39.433 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:51:40.435 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:51:41.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:41 smithi102 conmon[37544]: cluster 2024-01-13T09:51:40.719646+0000 mgr.smithi084.itrhok (mgr.14180) 204 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:41.779 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:41 smithi084 conmon[33408]: cluster 2024-01-13T09:51:40.719646+0000 mgr.smithi084.itrhok (mgr.14180) 204 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:42.432 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:51:43.110 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:51:43.486 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:43 smithi102 conmon[37544]: audit 2024-01-13T09:51:42.208591+0000 mon.smithi084 (mon.0) 529 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:51:43.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:43 smithi102 conmon[37544]: audit 2024-01-13T09:51:42.438748+0000 mon.smithi084 (mon.0) 530 : audit [DBG] from='client.? 172.21.15.84:0/4130310221' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:51:43.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:43 smithi102 conmon[37544]: cluster 2024-01-13T09:51:42.720054+0000 mgr.smithi084.itrhok (mgr.14180) 205 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:43.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:43 smithi084 conmon[33408]: audit 2024-01-13T09:51:42.208591+0000 mon.smithi084 (mon.0) 529 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:51:43.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:43 smithi084 conmon[33408]: audit 2024-01-13T09:51:42.438748+0000 mon.smithi084 (mon.0) 530 : audit [DBG] from='client.? 172.21.15.84:0/4130310221' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:51:43.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:43 smithi084 conmon[33408]: cluster 2024-01-13T09:51:42.720054+0000 mgr.smithi084.itrhok (mgr.14180) 205 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:44.111 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:51:44.975 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:44 smithi084 conmon[33408]: cluster 2024-01-13T09:51:44.720431+0000 mgr.smithi084.itrhok (mgr.14180) 206 2024-01-13T09:51:44.975 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:44 smithi084 conmon[33408]: : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:45.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:44 smithi102 conmon[37544]: cluster 2024-01-13T09:51:44.720431+0000 mgr.smithi084.itrhok (mgr.14180) 206 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:45.909 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:51:46.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:46 smithi102 conmon[37544]: audit 2024-01-13T09:51:45.916007+0000 mon.smithi084 (mon.0) 531 : audit [DBG] from='client.? 172.21.15.84:0/2706882950' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:51:46.567 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:46 smithi084 conmon[33408]: audit 2024-01-13T09:51:45.916007+0000 mon.smithi084 (mon.0) 531 : audit [DBG] from='client.? 172.21.15.84:0/2706882950' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:51:46.582 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:51:47.584 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:51:47.617 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:47 smithi084 conmon[33408]: cluster 2024-01-13T09:51:46.720770+0000 mgr.smithi084.itrhok (mgr.14180) 207 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:47.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:47 smithi102 conmon[37544]: cluster 2024-01-13T09:51:46.720770+0000 mgr.smithi084.itrhok (mgr.14180) 207 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:49.076 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:48 smithi084 conmon[33408]: cluster 2024-01-13T09:51:48.721157+0000 mgr.smithi084.itrhok (mgr.14180) 208 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:49.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:48 smithi102 conmon[37544]: cluster 2024-01-13T09:51:48.721157+0000 mgr.smithi084.itrhok (mgr.14180) 208 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:49.432 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:51:50.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:49 smithi102 conmon[37544]: audit 2024-01-13T09:51:49.439125+0000 mon.smithi084 (mon.0) 532 : audit [DBG] from='client.? 172.21.15.84:0/2307532879' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:51:50.265 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:49 smithi084 conmon[33408]: audit 2024-01-13T09:51:49.439125+0000 mon.smithi084 (mon.0) 532 : audit [DBG] 2024-01-13T09:51:50.265 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:49 smithi084 conmon[33408]: from='client.? 172.21.15.84:0/2307532879' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:51:50.284 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:51:51.288 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:51:51.722 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:51 smithi084 conmon[33408]: cluster 2024-01-13T09:51:50.721834+0000 mgr.smithi084.itrhok (mgr.14180) 209 : cluster 2024-01-13T09:51:51.723 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:51 smithi084 conmon[33408]: [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:51.736 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:51 smithi102 conmon[37544]: cluster 2024-01-13T09:51:50.721834+0000 mgr.smithi084.itrhok (mgr.14180) 209 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:52.916 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:52 smithi084 conmon[33408]: cluster 2024-01-13T09:51:52.722228+0000 mgr.smithi084.itrhok (mgr.14180) 210 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:53.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:52 smithi102 conmon[37544]: cluster 2024-01-13T09:51:52.722228+0000 mgr.smithi084.itrhok (mgr.14180) 210 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:53.270 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:51:53.947 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:51:54.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:53 smithi102 conmon[37544]: audit 2024-01-13T09:51:53.276975+0000 mon.smithi084 (mon.0) 533 : audit [DBG] from='client.? 172.21.15.84:0/3157920154' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:51:54.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:53 smithi084 conmon[33408]: audit 2024-01-13T09:51:53.276975+0000 mon.smithi084 (mon.0) 533 : audit [DBG] from='client.? 172.21.15.84:0/3157920154' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:51:54.947 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:51:55.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:54 smithi102 conmon[37544]: cluster 2024-01-13T09:51:54.722707+0000 mgr.smithi084.itrhok (mgr.14180) 211 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:55.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:54 smithi084 conmon[33408]: cluster 2024-01-13T09:51:54.722707 2024-01-13T09:51:55.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:54 smithi084 conmon[33408]: +0000 mgr.smithi084.itrhok (mgr.14180) 211 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:56.770 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:51:57.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:56 smithi102 conmon[37544]: cluster 2024-01-13T09:51:56.723211+0000 mgr.smithi084.itrhok (mgr.14180) 212 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:57.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:56 smithi102 conmon[37544]: audit 2024-01-13T09:51:57.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:56 smithi102 conmon[37544]: 2024-01-13T09:51:56.776924+0000 mon.smithi084 (mon.0) 534 : audit [DBG] from='client.? 172.21.15.84:0/689742502' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:51:57.279 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:56 smithi084 conmon[33408]: cluster 2024-01-13T09:51:56.723211+0000 mgr.smithi084.itrhok (mgr.14180) 212 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:57.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:56 smithi084 conmon[33408]: audit 2024-01-13T09:51:56.776924+0000 mon.smithi084 (mon.0) 534 : audit [DBG] from='client.? 172.21.15.84:0/689742502' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:51:57.504 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:51:58.505 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:51:59.050 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:58 smithi084 conmon[33408]: cluster 2024-01-13T09:51:58.723745+0000 mgr.smithi084.itrhok 2024-01-13T09:51:59.050 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:51:58 smithi084 conmon[33408]: (mgr.14180) 213 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:51:59.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:51:58 smithi102 conmon[37544]: cluster 2024-01-13T09:51:58.723745+0000 mgr.smithi084.itrhok (mgr.14180) 213 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:00.475 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:52:01.119 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:52:01.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:01 smithi084 conmon[33408]: audit 2024-01-13T09:52:00.482045+0000 mon.smithi084 (mon.0) 535 : audit [DBG] from='client.? 172.21.15.84:0/1435847761' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:52:01.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:01 smithi084 conmon[33408]: cluster 2024-01-13T09:52:00.724296+0000 mgr.smithi084.itrhok (mgr.14180) 214 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:01.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:01 smithi102 conmon[37544]: audit 2024-01-13T09:52:00.482045+0000 mon.smithi084 (mon.0) 535 : audit [DBG] from='client.? 172.21.15.84:0/1435847761' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:52:01.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:01 smithi102 conmon[37544]: 2024-01-13T09:52:01.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:01 smithi102 conmon[37544]: cluster 2024-01-13T09:52:00.724296+0000 mgr.smithi084.itrhok (mgr.14180) 214 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:02.120 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:52:03.021 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:02 smithi084 conmon[33408]: cluster 2024-01-13T09:52:03.021 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:02 smithi084 conmon[33408]: 2024-01-13T09:52:02.724694+0000 mgr.smithi084.itrhok (mgr.14180) 215 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:03.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:02 smithi102 conmon[37544]: cluster 2024-01-13T09:52:02.724694+0000 mgr.smithi084.itrhok (mgr.14180) 215 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:04.001 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:52:04.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:04 smithi084 conmon[33408]: audit 2024-01-13T09:52:04.007764+0000 mon.smithi084 ( 2024-01-13T09:52:04.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:04 smithi084 conmon[33408]: mon.0) 536 : audit [DBG] from='client.? 172.21.15.84:0/2939969523' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:52:04.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:04 smithi102 conmon[37544]: audit 2024-01-13T09:52:04.007764+0000 mon.smithi084 (mon.0) 536 : audit [DBG] from='client.? 172.21.15.84:0/2939969523' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:52:04.682 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:52:05.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:05 smithi102 conmon[37544]: cluster 2024-01-13T09:52:04.725320+0000 mgr.smithi084.itrhok (mgr.14180) 216 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:05.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:05 smithi084 conmon[33408]: cluster 2024-01-13T09:52:04.725320+0000 mgr.smithi084.itrhok (mgr.14180) 216 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:05.683 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:52:06.930 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:06 smithi084 conmon[33408]: cluster 2024-01-13T09:52:06.725801+0000 mgr.smithi084.itrhok (mgr.14180) 217 : cluster 2024-01-13T09:52:06.930 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:06 smithi084 conmon[33408]: [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:07.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:06 smithi102 conmon[37544]: cluster 2024-01-13T09:52:06.725801+0000 mgr.smithi084.itrhok (mgr.14180) 217 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:07.587 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:52:08.030 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:07 smithi084 conmon[33408]: audit 2024-01-13T09:52:07.593508+0000 mon.smithi084 (mon.0) 537 : audit [DBG] from='client.? 172.21.15.84:0/872699226' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:52:08.210 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:52:08.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:07 smithi102 conmon[37544]: audit 2024-01-13T09:52:07.593508+0000 mon.smithi084 (mon.0) 537 : audit [DBG] from='client.? 172.21.15.84:0/872699226' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:52:09.211 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:52:09.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:08 smithi102 conmon[37544]: cluster 2024-01-13T09:52:08.726187+0000 mgr.smithi084.itrhok (mgr.14180) 218 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:09.244 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:08 smithi084 conmon[33408]: cluster 2024-01-13T09:52:08.726187+0000 mgr.smithi084.itrhok (mgr.14180) 218 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:11.202 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:52:11.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:11 smithi084 conmon[33408]: cluster 2024-01-13T09:52:10.726592+0000 mgr.smithi084.itrhok (mgr.14180) 219 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:11.530 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:11 smithi084 conmon[33408]: audit 2024-01-13T09:52:11.208075+0000 mon.smithi084 (mon.0) 538 : audit [DBG] from='client.? 172.21.15.84:0/1632962582' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:52:11.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:11 smithi102 conmon[37544]: cluster 2024-01-13T09:52:10.726592+0000 mgr.smithi084.itrhok (mgr.14180) 219 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:11.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:11 smithi102 conmon[37544]: audit 2024-01-13T09:52:11.208075+0000 2024-01-13T09:52:11.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:11 smithi102 conmon[37544]: mon.smithi084 (mon.0) 538 : audit [DBG] from='client.? 172.21.15.84:0/1632962582' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:52:11.884 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:52:12.885 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:52:13.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:12 smithi102 conmon[37544]: cluster 2024-01-13T09:52:12.727043+0000 mgr.smithi084.itrhok (mgr.14180) 220 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:13.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:12 smithi084 conmon[33408]: cluster 2024-01-13T09:52:12.727043+0000 mgr.smithi084.itrhok (mgr.14180) 220 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:14.766 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:52:15.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:14 smithi102 conmon[37544]: cluster 2024-01-13T09:52:14.727437+0000 mgr.smithi084.itrhok (mgr.14180) 221 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:15.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:14 smithi102 conmon[37544]: audit 2024-01-13T09:52:14. 2024-01-13T09:52:15.238 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:14 smithi102 conmon[37544]: 772863+0000 mon.smithi084 (mon.0) 539 : audit [DBG] from='client.? 172.21.15.84:0/3784768531' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:52:15.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:14 smithi084 conmon[33408]: cluster 2024-01-13T09:52:14.727437+0000 mgr.smithi084.itrhok (mgr.14180) 221 : 2024-01-13T09:52:15.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:14 smithi084 conmon[33408]: cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:15.281 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:14 smithi084 conmon[33408]: audit 2024-01-13T09:52:14.772863+0000 mon.smithi084 (mon.0) 539 : audit [DBG] from='client.? 172.21.15.84:0/3784768531' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:52:15.418 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:52:16.419 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:52:17.112 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:16 smithi084 conmon[33408]: cluster 2024-01-13T09:52:16.727797+0000 mgr.smithi084.itrhok (mgr.14180) 222 : 2024-01-13T09:52:17.112 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:16 smithi084 conmon[33408]: cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:17.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:16 smithi102 conmon[37544]: cluster 2024-01-13T09:52:16.727797+0000 mgr.smithi084.itrhok (mgr.14180) 222 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:18.396 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:52:18.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:18 smithi102 conmon[37544]: audit 2024-01-13T09:52:18.403237+0000 mon.smithi084 (mon.0) 540 : audit [DBG] from='client.? 172.21.15.84:0/584031625' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:52:18.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:18 smithi084 conmon[33408]: audit 2024-01-13T09:52:18.403237+0000 mon.smithi084 (mon 2024-01-13T09:52:18.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:18 smithi084 conmon[33408]: .0) 540 : audit [DBG] from='client.? 172.21.15.84:0/584031625' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:52:19.043 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:52:19.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:19 smithi102 conmon[37544]: cluster 2024-01-13T09:52:18.728155+0000 mgr.smithi084.itrhok (mgr.14180) 223 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:19.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:19 smithi084 conmon[33408]: cluster 2024-01-13T09:52:18.728155+0000 mgr.smithi084.itrhok (mgr.14180) 223 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:20.044 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:52:21.626 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:21 smithi084 conmon[33408]: cluster 2024-01-13T09:52:20.728506+0000 mgr.smithi084.itrhok (mgr.14180) 224 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:21.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:21 smithi102 conmon[37544]: cluster 2024-01-13T09:52:20.728506+0000 mgr.smithi084.itrhok (mgr.14180) 224 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:22.011 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:52:22.694 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:22 smithi084 conmon[33408]: audit 2024-01-13T09:52:22.018168+0000 mon.smithi084 (mon.0) 541 : audit [DBG] from='client.? 172.21.15.84:0/3538741754' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:52:22.709 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:52:22.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:22 smithi102 conmon[37544]: audit 2024-01-13T09:52:22.018168+0000 mon.smithi084 (mon.0) 541 : audit [DBG] from='client.? 172.21.15.84:0/3538741754' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:52:23.710 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:52:23.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:23 smithi102 conmon[37544]: cluster 2024-01-13T09:52:22.728882+0000 mgr.smithi084.itrhok (mgr.14180) 225 : cluster [DBG] 2024-01-13T09:52:23.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:23 smithi102 conmon[37544]: pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:23.742 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:23 smithi084 conmon[33408]: cluster 2024-01-13T09:52:22.728882+0000 mgr.smithi084.itrhok (mgr.14180) 225 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:24.989 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:24 smithi084 conmon[33408]: cluster 2024-01-13T09:52:24.729264+0000 mgr.smithi084.itrhok (mgr.14180) 226 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:25.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:24 smithi102 conmon[37544]: cluster 2024-01-13T09:52:24.729264+0000 mgr.smithi084.itrhok (mgr.14180) 226 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:25.636 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:52:26.333 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:52:26.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:26 smithi102 conmon[37544]: audit 2024-01-13T09:52:25.643339+0000 mon.smithi084 (mon.0) 542 : audit [DBG] from='client.? 172.21.15.84:0/1122257159' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:52:26.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:26 smithi084 conmon[33408]: audit 2024-01-13T09:52:25.643339+0000 mon.smithi084 (mon.0) 542 : audit [DBG] from='client.? 172.21.15.84:0/1122257159' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:52:27.334 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:52:27.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:27 smithi102 conmon[37544]: cluster 2024-01-13T09:52:26.729917+0000 mgr.smithi084.itrhok (mgr.14180) 227 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:27.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:27 smithi084 conmon[33408]: cluster 2024-01-13T09:52:26.729917+0000 mgr.smithi084.itrhok (mgr.14180) 227 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:28.833 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:28 smithi084 conmon[33408]: cluster 2024-01-13T09:52:28.730553+0000 mgr.smithi084.itrhok (mgr.14180) 228 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:29.221 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:52:29.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:28 smithi102 conmon[37544]: cluster 2024-01-13T09:52:28.730553+0000 mgr.smithi084.itrhok (mgr.14180) 228 : cluster [DBG] 2024-01-13T09:52:29.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:28 smithi102 conmon[37544]: pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:29.824 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:52:30.237 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:29 smithi102 conmon[37544]: audit 2024-01-13T09:52:29.227984+0000 mon.smithi084 (mon.0) 543 : audit [DBG] from='client.? 172.21.15.84:0/3139488747' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:52:30.280 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:29 smithi084 conmon[33408]: audit 2024-01-13T09:52:29.227984+0000 mon.smithi084 (mon.0) 543 : audit [DBG] from='client.? 172.21.15.84:0/3139488747' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:52:30.825 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:52:31.567 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:31 smithi102 conmon[37544]: audit 2024-01-13T09:52:30.718677+0000 mon.smithi084 (mon.0) 544 : audit 2024-01-13T09:52:31.567 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:31 smithi102 conmon[37544]: [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T09:52:31.567 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:31 smithi102 conmon[37544]: cluster 2024-01-13T09:52:30.730949+0000 mgr.smithi084.itrhok (mgr.14180) 229 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:31.567 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:31 smithi102 conmon[37544]: audit 2024-01-13T09:52:31.052787+0000 mon.smithi084 (mon.0) 545 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:52:31.567 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:31 smithi102 conmon[37544]: audit 2024-01-13T09:52:31.053564+0000 mon.smithi084 (mon.0) 546 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T09:52:31.567 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:31 smithi102 conmon[37544]: audit 2024-01-13T09:52:31.057751+0000 mon.smithi084 (mon.0) 547 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:52:31.567 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:31 smithi102 conmon[37544]: audit 2024-01-13T09:52:31.061950+0000 mon.smithi084 (mon.0) 548 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T09:52:31.567 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:31 smithi102 conmon[37544]: audit 2024-01-13T09:52:31.065596+0000 mon.smithi084 (mon.0) 549 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:52:31.568 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:31 smithi102 conmon[37544]: audit 2024-01-13T09:52:31.066441+0000 mon.smithi084 (mon.0) 550 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:52:31.568 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:31 smithi102 conmon[37544]: audit 2024-01-13T09:52:31.066678+0000 mon.smithi084 (mon.0) 551 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:52:31.568 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:31 smithi102 conmon[37544]: audit 2024-01-13T09:52:31.067141+0000 mon.smithi084 (mon.0) 552 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:52:31.665 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:31 smithi084 conmon[33408]: audit 2024-01-13T09:52:30.718677+0000 mon.smithi084 (mon. 2024-01-13T09:52:31.665 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:31 smithi084 conmon[33408]: 0) 544 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T09:52:31.665 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:31 smithi084 conmon[33408]: cluster 2024-01-13T09:52:30.730949+0000 mgr.smithi084.itrhok (mgr.14180) 229 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:31.665 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:31 smithi084 conmon[33408]: audit 2024-01-13T09:52:31.052787+0000 mon.smithi084 (mon.0) 545 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:52:31.665 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:31 smithi084 conmon[33408]: audit 2024-01-13T09:52:31.053564+0000 mon.smithi084 (mon.0) 546 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T09:52:31.666 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:31 smithi084 conmon[33408]: audit 2024-01-13T09:52:31.057751+0000 mon.smithi084 (mon.0) 547 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:52:31.666 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:31 smithi084 conmon[33408]: audit 2024-01-13T09:52:31.061950+0000 mon.smithi084 (mon.0) 548 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T09:52:31.666 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:31 smithi084 conmon[33408]: audit 2024-01-13T09:52:31.065596+0000 mon.smithi084 (mon.0) 549 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:52:31.666 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:31 smithi084 conmon[33408]: audit 2024-01-13T09:52:31.066441+0000 mon.smithi084 (mon.0) 550 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:52:31.666 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:31 smithi084 conmon[33408]: audit 2024-01-13T09:52:31.066678+0000 mon.smithi084 (mon.0) 551 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:52:31.666 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:31 smithi084 conmon[33408]: audit 2024-01-13T09:52:31.067141+0000 mon.smithi084 (mon.0) 552 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:52:32.469 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:32 smithi084 conmon[33408]: cluster 2024-01-13T09:52:31.058154+0000 2024-01-13T09:52:32.469 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:32 smithi084 conmon[33408]: mgr.smithi084.itrhok (mgr.14180) 230 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:32.469 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:32 smithi084 conmon[33408]: cluster 2024-01-13T09:52:31.307374+0000 mon.smithi084 (mon.0) 553 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T09:52:32.469 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:32 smithi084 conmon[33408]: cluster 2024-01-13T09:52:31.307403+0000 mon.smithi084 (mon.0) 554 : cluster [INF] Cluster is now healthy 2024-01-13T09:52:32.560 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:32 smithi102 conmon[37544]: cluster 2024-01-13T09:52:31.058154+0000 2024-01-13T09:52:32.560 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:32 smithi102 conmon[37544]: mgr.smithi084.itrhok (mgr.14180) 230 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:32.560 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:32 smithi102 conmon[37544]: cluster 2024-01-13T09:52:31.307374+0000 mon.smithi084 (mon.0) 553 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T09:52:32.560 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:32 smithi102 conmon[37544]: cluster 2024-01-13T09:52:31.307403+0000 mon.smithi084 (mon.0) 554 : cluster [INF] Cluster is now healthy 2024-01-13T09:52:33.236 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:52:33.531 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:33 smithi084 conmon[33408]: audit 2024-01-13T09:52:33.242828+0000 mon.smithi084 (mon.0) 555 : audit [DBG] from='client.? 172.21.15.84:0/2517218635' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:52:33.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:33 smithi102 conmon[37544]: audit 2024-01-13T09:52:33.242828+0000 mon.smithi084 (mon.0) 555 : audit [DBG] from='client.? 172.21.15.84:0/2517218635' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:52:33.930 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:52:34.590 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:34 smithi084 conmon[33408]: cluster 2024-01-13T09:52:33.058380+0000 mgr.smithi084.itrhok (mgr.14180) 231 2024-01-13T09:52:34.590 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:34 smithi084 conmon[33408]: : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:34.591 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:34 smithi084 conmon[33408]: audit 2024-01-13T09:52:33.636326+0000 mon.smithi102 (mon.1) 12 : audit [INF] from='client.? 172.21.15.102:0/2992922980' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7e36d92-73ab-4aac-b6d9-e294ba5ab241"}]: dispatch 2024-01-13T09:52:34.591 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:34 smithi084 conmon[33408]: audit 2024-01-13T09:52:33.636885+0000 mon.smithi084 (mon.0) 556 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7e36d92-73ab-4aac-b6d9-e294ba5ab241"}]: dispatch 2024-01-13T09:52:34.591 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:34 smithi084 conmon[33408]: audit 2024-01-13T09:52:33.642999+0000 mon.smithi084 (mon.0) 557 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b7e36d92-73ab-4aac-b6d9-e294ba5ab241"}]': finished 2024-01-13T09:52:34.591 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:34 smithi084 conmon[33408]: cluster 2024-01-13T09:52:33.643049+0000 mon.smithi084 (mon.0) 558 : cluster [DBG] osdmap e26: 1 total, 0 up, 1 in 2024-01-13T09:52:34.591 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:34 smithi084 conmon[33408]: audit 2024-01-13T09:52:33.643167+0000 mon.smithi084 (mon.0) 559 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:52:34.591 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:34 smithi084 conmon[33408]: audit 2024-01-13T09:52:33.925460+0000 2024-01-13T09:52:34.592 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:34 smithi084 conmon[33408]: mon.smithi102 (mon.1) 13 : audit [INF] from='client.? 172.21.15.102:0/1035510580' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:52:34.592 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:34 smithi084 conmon[33408]: audit 2024-01-13T09:52:33.925955+0000 mon.smithi084 (mon.0) 560 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:52:34.592 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:34 smithi084 conmon[33408]: audit 2024-01-13T09:52:33.934359+0000 mon.smithi084 2024-01-13T09:52:34.592 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:34 smithi084 conmon[33408]: (mon.0) 561 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:52:34.592 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:34 smithi084 conmon[33408]: cluster 2024-01-13T09:52 2024-01-13T09:52:34.592 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:34 smithi084 conmon[33408]: :33.934413+0000 mon.smithi084 (mon.0) 562 : cluster 2024-01-13T09:52:34.592 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:34 smithi084 conmon[33408]: [DBG] osdmap e27: 0 total, 0 up, 0 in 2024-01-13T09:52:34.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:34 smithi102 conmon[37544]: cluster 2024-01-13T09:52:33.058380+0000 mgr.smithi084.itrhok (mgr.14180) 231 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:34.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:34 smithi102 conmon[37544]: audit 2024-01-13T09:52:33.636326+0000 mon.smithi102 (mon.1) 12 : audit [INF] from='client.? 172.21.15.102:0/2992922980' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7e36d92-73ab-4aac-b6d9-e294ba5ab241"}]: dispatch 2024-01-13T09:52:34.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:34 smithi102 conmon[37544]: audit 2024-01-13T09:52:33.636885+0000 mon.smithi084 (mon.0) 556 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7e36d92-73ab-4aac-b6d9-e294ba5ab241"}]: dispatch 2024-01-13T09:52:34.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:34 smithi102 conmon[37544]: audit 2024-01-13T09:52:33.642999+0000 mon.smithi084 (mon.0) 557 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b7e36d92-73ab-4aac-b6d9-e294ba5ab241"}]': finished 2024-01-13T09:52:34.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:34 smithi102 conmon[37544]: cluster 2024-01-13T09:52:33.643049+0000 mon.smithi084 (mon.0) 558 : cluster [DBG] osdmap e26: 1 total, 0 up, 1 in 2024-01-13T09:52:34.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:34 smithi102 conmon[37544]: audit 2024-01-13T09:52:33.643167+0000 mon.smithi084 (mon.0) 559 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:52:34.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:34 smithi102 conmon[37544]: audit 2024-01-13T09:52:33.925460+0000 mon.smithi102 (mon.1) 13 : audit [INF] from='client.? 172.21.15.102:0/1035510580' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:52:34.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:34 smithi102 conmon[37544]: audit 2024-01-13T09:52:33.925955+0000 mon.smithi084 (mon.0) 560 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:52:34.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:34 smithi102 conmon[37544]: audit 2024-01-13T09:52:33.934359+0000 mon.smithi084 (mon.0) 561 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:52:34.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:34 smithi102 conmon[37544]: cluster 2024-01-13T09:52:33.934413+0000 mon.smithi084 (mon.0) 562 : cluster [DBG] osdmap e27: 0 total, 0 up, 0 in 2024-01-13T09:52:34.931 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:52:35.610 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: cephadm 2024-01-13T09:52:34.596815+0000 mgr.smithi084.itrhok (mgr.14180) 232 : cephadm [ERR] executing create_from_spec_one(([('smithi084', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi102', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:52:35.610 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:52:35.610 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:52:35.611 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: return f(*arg) 2024-01-13T09:52:35.611 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:52:35.611 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:52:35.611 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:52:35.611 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: code, '\n'.join(err))) 2024-01-13T09:52:35.611 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:52:35.611 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmplkk8x_vc:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp7rre0t9o:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:52:35.611 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:52:35.611 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:52:35.611 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:52:35.611 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new b7e36d92-73ab-4aac-b6d9-e294ba5ab241 2024-01-13T09:52:35.611 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:52:35.612 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:52:35.612 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:52:35.612 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:52:35.612 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:52:35.612 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr self.prepare() 2024-01-13T09:52:35.612 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:52:35.612 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:52:35.612 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:52:35.612 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:52:35.612 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:52:35.612 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:52:35.612 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:52:35.613 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:52:35.613 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:52:35.613 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:52:35.613 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:52:35.613 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:52:35.613 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:52:35.613 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:52:35.613 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:52:35.613 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:52:35.613 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:52:35.613 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:52:35.613 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:52:35.613 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:52:35.613 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:52:35.614 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:52:35.614 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:52:35.614 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:52:35.614 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:52:35.614 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:52:35.614 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:52:35.614 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:52:35.614 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:52:35.614 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:52:35.614 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:52:35.614 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:52:35.614 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:52:35.614 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:52:35.614 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:52:35.615 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:52:35.615 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:52:35.615 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:52:35.615 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:52:35.615 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:52:35.615 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:52:35.615 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:52:35.615 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:52:35.615 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:52:35.615 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:52:35.615 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:52:35.615 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:52:35.615 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:52:35.616 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:52:35.616 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:52:35.616 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:52:35.616 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:52:35.616 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: main() 2024-01-13T09:52:35.616 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:52:35.616 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: r = ctx.func(ctx) 2024-01-13T09:52:35.616 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:52:35.616 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:52:35.616 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:52:35.616 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:52:35.616 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:52:35.617 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:52:35.617 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-8777 2024-01-13T09:52:35.617 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: 4f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:52:35.617 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:52:35.617 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:52:35.617 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:52:35.617 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:52:35.617 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:52:35.617 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmplkk8x_vc:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp7rre0t9o:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:52:35.617 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: audit 2024-01-13T09:52:35.134444+0000 mon.smithi084 (mon.0) 563 : audit [INF] from='client.? 172.21.15.84:0/1212268605' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2308467e-8a88-4ae6-858c-1a096bffa64b"}]: dispatch 2024-01-13T09:52:35.617 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: audit 2024-01-13T09:52:35.140721+0000 mon.smithi084 (mon.0) 564 : audit [INF] from='client.? 172.21.15.84:0/1212268605' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2308467e-8a88-4ae6-858c-1a096bffa64b"}]': finished 2024-01-13T09:52:35.617 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: cluster 2024-01-13T09:52:35.140763+0000 mon.smithi084 (mon.0) 565 : cluster [DBG] osdmap e28: 1 total, 0 up, 1 in 2024-01-13T09:52:35.617 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:35 smithi084 conmon[33408]: audit 2024-01-13T09:52:35.140859+0000 mon.smithi084 (mon.0) 566 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:52:35.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: cephadm 2024-01-13T09:52:34.596815+0000 mgr.smithi084.itrhok (mgr.14180) 232 : cephadm [ERR] executing create_from_spec_one(([('smithi084', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi102', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:52:35.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:52:35.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:52:35.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: return f(*arg) 2024-01-13T09:52:35.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:52:35.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:52:35.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:52:35.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: code, '\n'.join(err))) 2024-01-13T09:52:35.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:52:35.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmplkk8x_vc:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp7rre0t9o:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:52:35.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:52:35.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:52:35.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:52:35.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new b7e36d92-73ab-4aac-b6d9-e294ba5ab241 2024-01-13T09:52:35.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:52:35.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:52:35.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:52:35.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:52:35.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:52:35.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr self.prepare() 2024-01-13T09:52:35.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:52:35.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:52:35.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:52:35.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:52:35.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:52:35.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:52:35.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:52:35.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:52:35.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:52:35.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:52:35.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:52:35.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:52:35.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:52:35.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:52:35.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:52:35.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:52:35.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:52:35.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:52:35.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:52:35.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:52:35.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:52:35.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:52:35.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:52:35.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:52:35.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:52:35.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:52:35.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:52:35.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:52:35.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:52:35.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:52:35.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:52:35.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:52:35.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:52:35.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:52:35.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:52:35.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:52:35.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:52:35.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:52:35.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:52:35.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:52:35.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:52:35.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:52:35.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:52:35.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:52:35.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:52:35.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:52:35.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:52:35.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:52:35.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:52:35.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:52:35.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:52:35.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:52:35.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: main() 2024-01-13T09:52:35.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:52:35.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: r = ctx.func(ctx) 2024-01-13T09:52:35.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:52:35.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:52:35.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:52:35.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:52:35.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:52:35.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:52:35.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-8777 2024-01-13T09:52:35.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: 4f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:52:35.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:52:35.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:52:35.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:52:35.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:52:35.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:52:35.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmplkk8x_vc:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp7rre0t9o:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:52:35.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: audit 2024-01-13T09:52:35.134444+0000 mon.smithi084 (mon.0) 563 : audit [INF] from='client.? 172.21.15.84:0/1212268605' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2308467e-8a88-4ae6-858c-1a096bffa64b"}]: dispatch 2024-01-13T09:52:35.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: audit 2024-01-13T09:52:35.140721+0000 mon.smithi084 (mon.0) 564 : audit [INF] from='client.? 172.21.15.84:0/1212268605' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2308467e-8a88-4ae6-858c-1a096bffa64b"}]': finished 2024-01-13T09:52:35.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: cluster 2024-01-13T09:52:35.140763+0000 mon.smithi084 (mon.0) 565 : cluster [DBG] osdmap e28: 1 total, 0 up, 1 in 2024-01-13T09:52:35.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:35 smithi102 conmon[37544]: audit 2024-01-13T09:52:35.140859+0000 mon.smithi084 (mon.0) 566 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:52:36.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:36 smithi102 conmon[37544]: cluster 2024-01-13T09:52:35.058678+0000 mgr.smithi084.itrhok (mgr.14180) 233 : cluster 2024-01-13T09:52:36.487 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:36 smithi102 conmon[37544]: [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:36.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:36 smithi102 conmon[37544]: audit 2024-01-13T09:52:35.452409+0000 mon.smithi084 (mon.0) 567 : audit [INF] from='client.? 172.21.15.84:0/1495634961' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:52:36.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:36 smithi102 conmon[37544]: audit 2024 2024-01-13T09:52:36.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:36 smithi102 conmon[37544]: -01-13T09:52:35.456777+0000 mon.smithi084 (mon.0) 568 : audit [INF] from='client.? 172.21.15.84:0/1495634961' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:52:36.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:36 smithi102 conmon[37544]: cluster 2024-01-13T09:52: 2024-01-13T09:52:36.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:36 smithi102 conmon[37544]: 35.456827+0000 mon.smithi084 (mon.0) 569 : cluster 2024-01-13T09:52:36.488 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:36 smithi102 conmon[37544]: [DBG] osdmap e29: 0 total, 0 up, 0 in 2024-01-13T09:52:36.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:36 smithi084 conmon[33408]: cluster 2024-01-13T09:52:35.058678+0000 mgr.smithi084.itrhok (mgr.14180) 233 : cluster [DBG] 2024-01-13T09:52:36.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:36 smithi084 conmon[33408]: pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:36.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:36 smithi084 conmon[33408]: audit 2024-01-13T09:52:35.452409+0000 mon.smithi084 (mon.0) 567 : audit [INF] from='client.? 172.21.15.84:0/1495634961' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:52:36.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:36 smithi084 conmon[33408]: audit 2024-01-13T09:52:35.456777+0000 mon.smithi084 (mon.0) 568 : audit [INF] from='client.? 172.21.15.84:0/1495634961' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:52:36.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:36 smithi084 conmon[33408]: cluster 2024-01-13T09:52:35.456827+0000 mon.smithi084 (mon.0) 569 : cluster [DBG] osdmap e29: 0 total, 0 up, 0 in 2024-01-13T09:52:37.481 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:52:37.722 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:37 smithi084 conmon[33408]: cluster 2024-01-13T09:52:37.336761+0000 mon.smithi084 (mon.0) 570 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T09:52:37.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:37 smithi102 conmon[37544]: cluster 2024-01-13T09:52:37.336761+0000 mon.smithi084 (mon.0) 570 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T09:52:38.213 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:52:38.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: cephadm 2024-01-13T09:52:36.864013+0000 mgr.smithi084.itrhok (mgr.14180) 234 : cephadm [ERR] executing create_from_spec_one(([('smithi084', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi102', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:52:38.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:52:38.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:52:38.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: return f(*arg) 2024-01-13T09:52:38.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:52:38.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:52:38.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:52:38.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: code, '\n'.join(err))) 2024-01-13T09:52:38.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi084/config 2024-01-13T09:52:38.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpeu__c3x0:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp6yduj9em:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:52:38.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:52:38.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:52:38.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:52:38.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 2308467e-8a88-4ae6-858c-1a096bffa64b 2024-01-13T09:52:38.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:52:38.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:52:38.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:52:38.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:52:38.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:52:38.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr self.prepare() 2024-01-13T09:52:38.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:52:38.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:52:38.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:52:38.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:52:38.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:52:38.533 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:52:38.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:52:38.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:52:38.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:52:38.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:52:38.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:52:38.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:52:38.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:52:38.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:52:38.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:52:38.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:52:38.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:52:38.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:52:38.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:52:38.534 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:52:38.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:52:38.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:52:38.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:52:38.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:52:38.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:52:38.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:52:38.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:52:38.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:52:38.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:52:38.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:52:38.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:52:38.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:52:38.535 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:52:38.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:52:38.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:52:38.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:52:38.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:52:38.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:52:38.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:52:38.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:52:38.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:52:38.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:52:38.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:52:38.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:52:38.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:52:38.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:52:38.536 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:52:38.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:52:38.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:52:38.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:52:38.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:52:38.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:52:38.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: main() 2024-01-13T09:52:38.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:52:38.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: r = ctx.func(ctx) 2024-01-13T09:52:38.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:52:38.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:52:38.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:52:38.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:52:38.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:52:38.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:52:38.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-8777 2024-01-13T09:52:38.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: 4f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:52:38.537 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:52:38.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:52:38.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:52:38.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:52:38.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:52:38.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpeu__c3x0:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp6yduj9em:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:52:38.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: cephadm 2024-01-13T09:52:36.865284+0000 mgr.smithi084.itrhok (mgr.14180) 235 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T09:52:38.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: service_id: all-available-devices 2024-01-13T09:52:38.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: service_name: osd.all-available-devices 2024-01-13T09:52:38.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: placement: 2024-01-13T09:52:38.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: host_pattern: '*' 2024-01-13T09:52:38.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: spec: 2024-01-13T09:52:38.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: data_devices: 2024-01-13T09:52:38.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: all: true 2024-01-13T09:52:38.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: filter_logic: AND 2024-01-13T09:52:38.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: objectstore: bluestore 2024-01-13T09:52:38.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:52:38.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmplkk8x_vc:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp7rre0t9o:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:52:38.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:52:38.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:52:38.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:52:38.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new b7e36d92-73ab-4aac-b6d9-e294ba5ab241 2024-01-13T09:52:38.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:52:38.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:52:38.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:52:38.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:52:38.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:52:38.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr self.prepare() 2024-01-13T09:52:38.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:52:38.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:52:38.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:52:38.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:52:38.539 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:52:38.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:52:38.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:52:38.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:52:38.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:52:38.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:52:38.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:52:38.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:52:38.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:52:38.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:52:38.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:52:38.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:52:38.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:52:38.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:52:38.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:52:38.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:52:38.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:52:38.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:52:38.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:52:38.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:52:38.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:52:38.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:52:38.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:52:38.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:52:38.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:52:38.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:52:38.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:52:38.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:52:38.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:52:38.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:52:38.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:52:38.541 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:52:38.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:52:38.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:52:38.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:52:38.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:52:38.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:52:38.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:52:38.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:52:38.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:52:38.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:52:38.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/ 2024-01-13T09:52:38.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: lvm/zap.py", line 302, in zap_osd 2024-01-13T09:52:38.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:52:38.542 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:52:38.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:52:38.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:52:38.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:52:38.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:52:38.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: main() 2024-01-13T09:52:38.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:52:38.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: r = ctx.func(ctx) 2024-01-13T09:52:38.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:52:38.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:52:38.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:52:38.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:52:38.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:52:38.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:52:38.543 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:52:38.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:52:38.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:52:38.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:52:38.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:52:38.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:52:38.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmplkk8x_vc:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp7rre0t9o:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:52:38.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:52:38.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T09:52:38.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: if self._apply_service(spec): 2024-01-13T09:52:38.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T09:52:38.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T09:52:38.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T09:52:38.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T09:52:38.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T09:52:38.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T09:52:38.545 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T09:52:38.545 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T09:52:38.545 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T09:52:38.545 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: raise self._value 2024-01-13T09:52:38.545 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T09:52:38.545 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: result = (True, func(*args, **kwds)) 2024-01-13T09:52:38.545 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T09:52:38.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: return list(map(*args)) 2024-01-13T09:52:38.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:52:38.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: return f(*arg) 2024-01-13T09:52:38.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:52:38.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:52:38.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:52:38.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: code, '\n'.join(err))) 2024-01-13T09:52:38.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:52:38.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmplkk8x_vc:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp7rre0t9o:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:52:38.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:52:38.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:52:38.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:52:38.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new b7e36d92-73ab-4aac-b6d9-e294ba5ab241 2024-01-13T09:52:38.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:52:38.546 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:52:38.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:52:38.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:52:38.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:52:38.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr self.prepare() 2024-01-13T09:52:38.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:52:38.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:52:38.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:52:38.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:52:38.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:52:38.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:52:38.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:52:38.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:52:38.547 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:52:38.548 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:52:38.548 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:52:38.548 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:52:38.548 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:52:38.548 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:52:38.548 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:52:38.548 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:52:38.548 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:52:38.548 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:52:38.548 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:52:38.548 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:52:38.548 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:52:38.548 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:52:38.548 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:52:38.549 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:52:38.549 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:52:38.549 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:52:38.549 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:52:38.549 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:52:38.549 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:52:38.549 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:52:38.549 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:52:38.549 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:52:38.549 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:52:38.549 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:52:38.549 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:52:38.549 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:52:38.549 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:52:38.549 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:52:38.550 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:52:38.550 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:52:38.550 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:52:38.550 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:52:38.550 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:52:38.550 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:52:38.550 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:52:38.550 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:52:38.550 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:52:38.550 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:52:38.550 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:52:38.550 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:52:38.550 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:52:38.550 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:52:38.551 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:52:38.551 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: main() 2024-01-13T09:52:38.551 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:52:38.551 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: r = ctx.func(ctx) 2024-01-13T09:52:38.551 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:52:38.551 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:52:38.551 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:52:38.551 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:52:38.551 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:52:38.551 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:52:38.551 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:52:38.551 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:52:38.551 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:52:38.551 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:52:38.551 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:52:38.551 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:52:38.552 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmplkk8x_vc:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp7rre0t9o:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:52:38.552 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: cluster 2024-01-13T09:52:36.865928+0000 mgr.smithi084.itrhok (mgr.14180) 236 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:38.552 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:38 smithi084 conmon[33408]: audit 2024-01-13T09:52:37.487180+0000 mon.smithi084 (mon.0) 571 : audit [DBG] from='client.? 172.21.15.84:0/2846518819' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:52:38.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: cephadm 2024-01-13T09:52:36.864013+0000 mgr.smithi084.itrhok (mgr.14180) 234 : cephadm [ERR] executing create_from_spec_one(([('smithi084', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi102', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:52:38.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:52:38.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:52:38.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: return f(*arg) 2024-01-13T09:52:38.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:52:38.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:52:38.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:52:38.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: code, '\n'.join(err))) 2024-01-13T09:52:38.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi084/config 2024-01-13T09:52:38.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpeu__c3x0:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp6yduj9em:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:52:38.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:52:38.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:52:38.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:52:38.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 2308467e-8a88-4ae6-858c-1a096bffa64b 2024-01-13T09:52:38.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:52:38.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:52:38.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:52:38.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:52:38.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:52:38.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr self.prepare() 2024-01-13T09:52:38.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:52:38.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:52:38.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:52:38.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:52:38.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:52:38.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:52:38.741 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:52:38.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:52:38.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:52:38.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:52:38.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:52:38.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:52:38.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:52:38.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:52:38.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:52:38.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:52:38.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:52:38.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:52:38.742 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:52:38.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:52:38.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:52:38.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:52:38.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:52:38.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:52:38.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:52:38.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:52:38.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:52:38.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:52:38.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:52:38.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:52:38.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:52:38.743 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:52:38.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:52:38.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:52:38.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:52:38.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:52:38.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:52:38.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:52:38.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:52:38.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:52:38.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:52:38.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:52:38.744 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:52:38.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:52:38.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:52:38.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:52:38.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:52:38.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:52:38.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:52:38.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:52:38.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:52:38.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:52:38.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: main() 2024-01-13T09:52:38.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:52:38.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: r = ctx.func(ctx) 2024-01-13T09:52:38.745 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:52:38.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:52:38.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:52:38.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:52:38.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:52:38.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:52:38.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-8777 2024-01-13T09:52:38.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: 4f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:52:38.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:52:38.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:52:38.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:52:38.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:52:38.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:52:38.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpeu__c3x0:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp6yduj9em:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:52:38.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: cephadm 2024-01-13T09:52:36.865284+0000 mgr.smithi084.itrhok (mgr.14180) 235 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T09:52:38.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: service_id: all-available-devices 2024-01-13T09:52:38.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: service_name: osd.all-available-devices 2024-01-13T09:52:38.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: placement: 2024-01-13T09:52:38.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: host_pattern: '*' 2024-01-13T09:52:38.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: spec: 2024-01-13T09:52:38.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: data_devices: 2024-01-13T09:52:38.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: all: true 2024-01-13T09:52:38.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: filter_logic: AND 2024-01-13T09:52:38.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: objectstore: bluestore 2024-01-13T09:52:38.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:52:38.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmplkk8x_vc:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp7rre0t9o:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:52:38.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:52:38.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:52:38.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:52:38.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new b7e36d92-73ab-4aac-b6d9-e294ba5ab241 2024-01-13T09:52:38.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:52:38.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:52:38.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:52:38.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:52:38.748 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:52:38.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr self.prepare() 2024-01-13T09:52:38.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:52:38.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:52:38.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:52:38.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:52:38.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:52:38.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:52:38.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:52:38.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:52:38.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:52:38.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:52:38.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:52:38.749 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:52:38.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:52:38.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:52:38.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:52:38.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:52:38.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:52:38.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:52:38.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:52:38.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:52:38.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:52:38.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:52:38.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:52:38.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:52:38.750 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:52:38.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:52:38.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:52:38.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:52:38.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:52:38.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:52:38.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:52:38.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:52:38.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:52:38.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:52:38.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:52:38.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:52:38.751 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:52:38.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:52:38.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:52:38.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:52:38.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:52:38.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:52:38.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:52:38.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:52:38.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:52:38.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/ 2024-01-13T09:52:38.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: lvm/zap.py", line 302, in zap_osd 2024-01-13T09:52:38.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:52:38.752 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:52:38.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:52:38.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:52:38.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:52:38.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:52:38.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: main() 2024-01-13T09:52:38.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:52:38.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: r = ctx.func(ctx) 2024-01-13T09:52:38.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:52:38.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:52:38.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:52:38.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:52:38.753 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:52:38.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:52:38.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:52:38.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:52:38.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:52:38.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:52:38.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:52:38.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:52:38.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmplkk8x_vc:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp7rre0t9o:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:52:38.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:52:38.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T09:52:38.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: if self._apply_service(spec): 2024-01-13T09:52:38.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T09:52:38.754 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T09:52:38.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T09:52:38.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T09:52:38.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T09:52:38.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T09:52:38.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T09:52:38.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T09:52:38.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T09:52:38.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: raise self._value 2024-01-13T09:52:38.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T09:52:38.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: result = (True, func(*args, **kwds)) 2024-01-13T09:52:38.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T09:52:38.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: return list(map(*args)) 2024-01-13T09:52:38.755 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:52:38.756 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: return f(*arg) 2024-01-13T09:52:38.756 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:52:38.756 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:52:38.756 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:52:38.756 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: code, '\n'.join(err))) 2024-01-13T09:52:38.756 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:52:38.756 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmplkk8x_vc:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp7rre0t9o:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:52:38.756 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:52:38.756 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:52:38.756 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:52:38.756 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new b7e36d92-73ab-4aac-b6d9-e294ba5ab241 2024-01-13T09:52:38.756 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:52:38.757 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:52:38.757 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:52:38.757 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:52:38.757 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:52:38.757 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr self.prepare() 2024-01-13T09:52:38.757 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:52:38.757 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:52:38.757 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:52:38.757 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:52:38.757 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:52:38.757 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:52:38.757 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:52:38.758 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:52:38.758 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:52:38.758 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:52:38.758 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:52:38.758 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:52:38.758 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:52:38.758 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:52:38.758 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:52:38.758 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:52:38.759 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:52:38.759 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:52:38.759 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:52:38.759 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:52:38.759 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:52:38.759 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:52:38.759 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:52:38.759 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:52:38.759 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:52:38.759 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:52:38.759 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:52:38.759 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:52:38.760 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:52:38.760 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:52:38.760 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:52:38.760 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:52:38.760 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:52:38.760 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:52:38.760 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:52:38.760 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:52:38.760 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:52:38.760 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:52:38.760 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:52:38.760 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:52:38.761 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:52:38.761 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:52:38.761 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:52:38.761 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:52:38.761 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:52:38.762 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:52:38.762 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:52:38.762 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:52:38.762 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:52:38.762 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:52:38.762 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:52:38.762 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:52:38.762 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:52:38.762 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: main() 2024-01-13T09:52:38.762 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:52:38.762 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: r = ctx.func(ctx) 2024-01-13T09:52:38.763 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:52:38.763 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:52:38.763 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:52:38.763 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:52:38.763 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:52:38.763 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:52:38.763 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:52:38.763 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:52:38.763 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:52:38.763 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:52:38.763 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:52:38.763 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:52:38.763 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmplkk8x_vc:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp7rre0t9o:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:52:38.764 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: cluster 2024-01-13T09:52:36.865928+0000 mgr.smithi084.itrhok (mgr.14180) 236 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:38.764 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:38 smithi102 conmon[37544]: audit 2024-01-13T09:52:37.487180+0000 mon.smithi084 (mon.0) 571 : audit [DBG] from='client.? 172.21.15.84:0/2846518819' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:52:39.214 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:52:40.681 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:40 smithi084 conmon[33408]: cluster 2024-01-13T09:52:38.866584+0000 mgr.smithi084.itrhok (mgr.14180) 237 : cluster 2024-01-13T09:52:40.681 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:40 smithi084 conmon[33408]: [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:40.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:40 smithi102 conmon[37544]: cluster 2024-01-13T09:52:38.866584+0000 mgr.smithi084.itrhok (mgr.14180) 237 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:41.072 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:52:41.732 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:41 smithi084 conmon[33408]: audit 2024-01-13T09:52:41.078437+0000 mon.smithi084 (mon.0) 572 : audit [DBG] from='client.? 172.21.15.84:0/1218632422' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:52:41.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:41 smithi102 conmon[37544]: audit 2024-01-13T09:52:41.078437+0000 mon.smithi084 (mon.0) 572 : audit [DBG] from='client.? 172.21.15.84:0/1218632422' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:52:41.748 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:52:42.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:42 smithi102 conmon[37544]: cluster 2024-01-13T09:52:40.866954+0000 mgr.smithi084.itrhok (mgr.14180) 238 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:42.749 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:52:42.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:42 smithi084 conmon[33408]: cluster 2024-01-13T09:52:40.866954+0000 mgr.smithi084.itrhok ( 2024-01-13T09:52:42.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:42 smithi084 conmon[33408]: mgr.14180) 238 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:44.710 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:52:44.734 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:44 smithi084 conmon[33408]: cluster 2024-01-13T09:52:42.867347+0000 mgr.smithi084.itrhok (mgr.14180) 239 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:44.737 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:44 smithi102 conmon[37544]: cluster 2024-01-13T09:52:42.867347+0000 mgr.smithi084.itrhok (mgr.14180) 239 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:45.353 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:52:45.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:45 smithi102 conmon[37544]: audit 2024-01-13T09:52:44.716157+0000 mon.smithi084 (mon.0) 573 : audit 2024-01-13T09:52:45.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:45 smithi102 conmon[37544]: [DBG] from='client.? 172.21.15.84:0/2770582770' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:52:45.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:45 smithi084 conmon[33408]: audit 2024-01-13T09:52:44.716157+0000 mon.smithi084 (mon.0) 573 : audit [DBG] from='client.? 172.21.15.84:0/2770582770' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:52:46.354 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:52:46.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:46 smithi102 conmon[37544]: cluster 2024-01-13T09:52:44.867736+0000 mgr.smithi084.itrhok (mgr.14180) 240 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:46.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:46 smithi084 conmon[33408]: cluster 2024-01-13T09:52:44.867736+0000 mgr.smithi084.itrhok (mgr.14180) 240 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:48.295 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:52:48.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:48 smithi102 conmon[37544]: cluster 2024-01-13T09:52:46.868121+0000 mgr.smithi084.itrhok (mgr.14180) 241 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:48.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:48 smithi102 conmon[37544]: audit 2024-01-13T09:52:48.301100+0000 mon.smithi084 (mon.0) 574 : audit [DBG] from='client.? 172.21.15.84:0/501869701' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:52:48.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:48 smithi084 conmon[33408]: cluster 2024-01-13T09:52:46.868121+0000 mgr.smithi084.itrhok (mgr.14180) 241 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:48.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:48 smithi084 conmon[33408]: audit 2024-01-13T09:52:48.301100+0000 mon.smithi084 (mon.0) 574 : audit [DBG] from='client.? 172.21.15.84:0/501869701' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:52:48.943 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:52:49.944 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:52:50.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:50 smithi102 conmon[37544]: cluster 2024-01-13T09:52:48.868493+0000 mgr.smithi084.itrhok (mgr.14180) 242 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:50.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:50 smithi084 conmon[33408]: cluster 2024-01-13T09:52:48.868493+0000 mgr.smithi084.itrhok (mgr.14180) 242 : cluster [DBG] 2024-01-13T09:52:50.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:50 smithi084 conmon[33408]: pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:51.851 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:52:52.586 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:52:52.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:52 smithi102 conmon[37544]: cluster 2024-01-13T09:52:50.868914+0000 mgr.smithi084.itrhok (mgr.14180) 243 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:52.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:52 smithi102 conmon[37544]: audit 2024-01-13T09:52:51.857606+0000 mon.smithi084 (mon.0) 575 : audit [DBG] from='client.? 172.21.15.84:0/3292458824' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:52:52.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:52 smithi084 conmon[33408]: cluster 2024-01-13T09:52:50.868914+0000 mgr.smithi084.itrhok (mgr.14180) 243 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:52.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:52 smithi084 conmon[33408]: audit 2024-01-13T09:52:51.857606+0000 mon.smithi084 (mon.0) 575 : audit [DBG] from='client.? 172.21.15.84:0/3292458824' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:52:53.587 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:52:54.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:54 smithi102 conmon[37544]: cluster 2024-01-13T09:52:52.869341+0000 mgr.smithi084.itrhok (mgr.14180) 244 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:54.780 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:54 smithi084 conmon[33408]: cluster 2024-01-13T09:52:52.869341+0000 mgr.smithi084.itrhok (mgr.14180) 2024-01-13T09:52:54.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:54 smithi084 conmon[33408]: 244 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:55.574 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:52:56.212 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:52:56.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:56 smithi102 conmon[37544]: cluster 2024-01-13T09:52:54.869717+0000 mgr.smithi084.itrhok (mgr.14180) 245 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:56.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:56 smithi102 conmon[37544]: audit 2024-01-13T09:52:55.580527+0000 mon.smithi084 (mon.0) 576 : audit [DBG] from='client.? 172.21.15.84:0/758527544' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:52:56.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:56 smithi084 conmon[33408]: cluster 2024-01-13T09:52:54.869717+0000 mgr.smithi084.itrhok (mgr.14180) 245 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:56.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:56 smithi084 conmon[33408]: audit 2024-01-13T09:52:55.580527+0000 mon.smithi084 (mon.0) 576 : audit [DBG] from='client.? 172.21.15.84:0/758527544' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:52:57.214 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:52:58.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:58 smithi102 conmon[37544]: cluster 2024-01-13T09:52:56.870181+0000 mgr.smithi084.itrhok (mgr.14180) 246 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:58.754 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:58 smithi084 conmon[33408]: cluster 2024-01-13T09:52:56.870181+0000 mgr.smithi084.itrhok (mgr.14180 2024-01-13T09:52:58.754 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:58 smithi084 conmon[33408]: ) 246 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:52:59.155 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:52:59.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:59 smithi102 conmon[37544]: audit 2024-01-13T09:52:59.161145+0000 mon.smithi084 (mon.0) 577 : audit [DBG] 2024-01-13T09:52:59.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:52:59 smithi102 conmon[37544]: from='client.? 172.21.15.84:0/3371978493' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:52:59.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:52:59 smithi084 conmon[33408]: audit 2024-01-13T09:52:59.161145+0000 mon.smithi084 (mon.0) 577 : audit [DBG] from='client.? 172.21.15.84:0/3371978493' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:52:59.849 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:53:00.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:00 smithi102 conmon[37544]: cluster 2024-01-13T09:52:58.870627+0000 mgr.smithi084.itrhok (mgr.14180) 247 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:00.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:00 smithi102 conmon[37544]: 2024-01-13T09:53:00.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:00 smithi084 conmon[33408]: cluster 2024-01-13T09:52:58.870627+0000 mgr.smithi084.itrhok (mgr.14180) 247 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:00.849 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:53:02.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:02 smithi102 conmon[37544]: cluster 2024-01-13T09:53:00.871238+0000 mgr.smithi084.itrhok (mgr.14180) 248 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:02.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:02 smithi084 conmon[33408]: cluster 2024-01-13T09:53:00.871238+0000 mgr.smithi084.itrhok (mgr.14180) 248 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:02.857 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:53:03.540 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:53:03.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:03 smithi102 conmon[37544]: audit 2024-01-13T09:53:02.862691+0000 mon.smithi084 (mon.0) 578 : audit [DBG] from='client.? 172.21.15.84:0/93276784' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:53:03.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:03 smithi084 conmon[33408]: audit 2024-01-13T09:53:02.862691+0000 mon.smithi084 (mon.0) 578 : audit [DBG] from='client.? 172.21.15.84:0/93276784' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:53:04.541 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:53:04.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:04 smithi102 conmon[37544]: cluster 2024-01-13T09:53:02.871666+0000 mgr.smithi084.itrhok (mgr.14180) 249 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:04.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:04 smithi084 conmon[33408]: cluster 2024-01-13T09:53:02.871666+0000 mgr.smithi084.itrhok (mgr.14180) 249 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:06.495 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:53:06.733 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:06 smithi084 conmon[33408]: cluster 2024-01-13T09:53:04.872040+0000 mgr.smithi084.itrhok (mgr.14180) 250 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:06.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:06 smithi102 conmon[37544]: cluster 2024-01-13T09:53:04.872040+0000 mgr.smithi084.itrhok (mgr.14180) 250 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:06.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:06 smithi102 conmon[37544]: 2024-01-13T09:53:07.165 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:53:07.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:07 smithi102 conmon[37544]: audit 2024-01-13T09:53:06.500633+0000 mon.smithi084 (mon.0) 579 : audit [DBG] from='client.? 172.21.15.84:0/854132842' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:53:07.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:07 smithi084 conmon[33408]: audit 2024-01-13T09:53:06.500633+0000 mon.smithi084 (mon.0) 579 : audit [DBG] from='client.? 172.21.15.84:0/854132842' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:53:08.166 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:53:08.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:08 smithi084 conmon[33408]: cluster 2024-01-13T09:53:06.872547+0000 mgr.smithi084.itrhok (mgr.14180) 251 : cluster [DBG] 2024-01-13T09:53:08.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:08 smithi084 conmon[33408]: pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:08.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:08 smithi102 conmon[37544]: cluster 2024-01-13T09:53:06.872547+0000 mgr.smithi084.itrhok (mgr.14180) 251 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:10.043 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:53:10.639 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:53:10.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:10 smithi084 conmon[33408]: cluster 2024-01-13T09:53:08.872904+0000 mgr.smithi084.itrhok (mgr.14180) 252 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:10.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:10 smithi084 conmon[33408]: audit 2024-01-13T09:53:10.048365+0000 mon.smithi084 (mon.0) 580 : audit [DBG] from='client.? 172.21.15.84:0/540657965' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:53:10.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:10 smithi102 conmon[37544]: cluster 2024-01-13T09:53:08.872904+0000 mgr.smithi084.itrhok (mgr.14180) 252 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:10.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:10 smithi102 conmon[37544]: audit 2024-01-13T09:53:10.048365+0000 mon.smithi084 (mon.0) 580 : audit [DBG] from='client.? 172.21.15.84:0/540657965' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:53:11.640 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:53:12.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:12 smithi084 conmon[33408]: cluster 2024-01-13T09:53:10.873424+0000 mgr.smithi084.itrhok (mgr.14180) 253 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:12.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:12 smithi102 conmon[37544]: cluster 2024-01-13T09:53:10.873424+0000 mgr.smithi084.itrhok (mgr.14180) 253 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:13.584 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:53:14.196 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:53:14.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:14 smithi084 conmon[33408]: cluster 2024-01-13T09:53:12.874046+0000 mgr.smithi084.itrhok (mgr.14180) 254 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:14.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:14 smithi084 conmon[33408]: audit 2024-01-13T09:53:13.590013+0000 mon.smithi084 (mon.0) 581 : audit [DBG] from='client.? 172.21.15.84:0/2928388707' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:53:14.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:14 smithi102 conmon[37544]: cluster 2024-01-13T09:53:12.874046+0000 mgr.smithi084.itrhok (mgr.14180) 254 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:14.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:14 smithi102 conmon[37544]: audit 2024-01-13T09:53:13.590013+0000 mon.smithi084 (mon.0) 581 : audit [DBG] from='client.? 172.21.15.84:0/2928388707' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:53:15.198 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:53:16.763 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:16 smithi084 conmon[33408]: cluster 2024-01-13T09:53:14.874654+0000 mgr.smithi084.itrhok (mgr.14180) 255 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:16.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:16 smithi102 conmon[37544]: cluster 2024-01-13T09:53:14.874654+0000 mgr.smithi084.itrhok (mgr.14180) 255 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:17.146 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:53:17.764 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:17 smithi084 conmon[33408]: audit 2024-01-13T09:53:17.151562+0000 mon.smithi084 (mon.0) 582 : audit [DBG] from='client.? 172.21.15.84:0/2253354356' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:53:17.779 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:53:17.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:17 smithi102 conmon[37544]: audit 2024-01-13T09:53:17.151562+0000 mon.smithi084 (mon.0) 582 : audit [DBG] from='client.? 172.21.15.84:0/2253354356' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:53:18.780 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:53:18.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:18 smithi084 conmon[33408]: cluster 2024-01-13T09:53:16.875022+0000 mgr.smithi084.itrhok (mgr.14180) 256 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:18.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:18 smithi102 conmon[37544]: cluster 2024-01-13T09:53:16.875022+0000 mgr.smithi084.itrhok (mgr.14180) 256 2024-01-13T09:53:18.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:18 smithi102 conmon[37544]: : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:20.737 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:53:20.763 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:20 smithi084 conmon[33408]: cluster 2024-01-13T09:53:18.875435+0000 mgr.smithi084.itrhok (mgr.14180) 257 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:20.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:20 smithi102 conmon[37544]: cluster 2024-01-13T09:53:18.875435+0000 mgr.smithi084.itrhok (mgr.14180) 257 : cluster [DBG] 2024-01-13T09:53:20.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:20 smithi102 conmon[37544]: pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:21.353 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:53:21.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:21 smithi084 conmon[33408]: audit 2024-01-13T09:53:20.743074+0000 mon.smithi084 (mon.0) 583 : audit [DBG] 2024-01-13T09:53:21.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:21 smithi084 conmon[33408]: from='client.? 172.21.15.84:0/2647700210' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:53:21.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:21 smithi102 conmon[37544]: audit 2024-01-13T09:53:20.743074+0000 mon.smithi084 (mon.0) 583 : audit [DBG] from='client.? 172.21.15.84:0/2647700210' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:53:22.354 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:53:22.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:22 smithi084 conmon[33408]: cluster 2024-01-13T09:53:20.875879+0000 mgr.smithi084.itrhok (mgr.14180) 258 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:22.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:22 smithi102 conmon[37544]: cluster 2024-01-13T09:53:20.875879+0000 mgr.smithi084.itrhok (mgr.14180) 258 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:24.181 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:53:24.793 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:24 smithi084 conmon[33408]: cluster 2024-01-13T09:53:22.876299+0000 mgr.smithi084.itrhok (mgr.14180) 259 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:24.793 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:24 smithi084 conmon[33408]: audit 2024-01-13T09:53:24.186745+0000 mon.smithi084 (mon.0) 584 : audit [DBG] from='client.? 172.21.15.84:0/3222413740' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:53:24.794 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:53:24.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:24 smithi102 conmon[37544]: cluster 2024-01-13T09:53:22.876299+0000 mgr.smithi084.itrhok (mgr.14180) 259 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:24.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:24 smithi102 conmon[37544]: audit 2024-01-13T09:53:24.186745+0000 mon.smithi084 (mon.0) 584 : audit [DBG] from='client.? 172.21.15.84:0/3222413740' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:53:25.795 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:53:26.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:26 smithi102 conmon[37544]: cluster 2024-01-13T09:53:24.876851+0000 mgr.smithi084.itrhok (mgr.14180) 260 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:27.022 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:26 smithi084 conmon[33408]: cluster 2024-01-13T09:53:24.876851+0000 mgr.smithi084.itrhok (mgr.14180) 260 : cluster [DBG] 2024-01-13T09:53:27.022 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:26 smithi084 conmon[33408]: pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:27.639 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:53:28.279 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:53:28.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:28 smithi102 conmon[37544]: cluster 2024-01-13T09:53:26.877492+0000 mgr.smithi084.itrhok (mgr.14180) 261 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:28.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:28 smithi102 conmon[37544]: audit 2024-01-13T09:53:27.645180+0000 mon.smithi084 (mon.0) 585 : audit [DBG] from='client.? 172.21.15.84:0/2381941540' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:53:29.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:28 smithi084 conmon[33408]: cluster 2024-01-13T09:53:26.877492+0000 mgr.smithi084.itrhok (mgr.14180) 261 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:29.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:28 smithi084 conmon[33408]: audit 2024-01-13T09:53:27.645180+0000 mon.smithi084 (mon.0) 585 : audit [DBG] from='client.? 172.21.15.84:0/2381941540' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:53:29.280 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:53:30.937 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:30 smithi084 conmon[33408]: cluster 2024-01-13T09:53:28.877971+0000 mgr.smithi084.itrhok (mgr.14180) 262 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:30.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:30 smithi102 conmon[37544]: cluster 2024-01-13T09:53:28.877971+0000 mgr.smithi084.itrhok (mgr.14180) 262 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:31.352 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:53:31.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:31 smithi084 conmon[33408]: audit 2024-01-13T09:53:31.357783+0000 mon.smithi084 (mon.0) 586 : audit [DBG] from='client.? 172.21.15.84:0/2038543439' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:53:31.951 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:53:31.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:31 smithi102 conmon[37544]: audit 2024-01-13T09:53:31.357783+0000 mon.smithi084 (mon.0) 586 : audit [DBG] from='client.? 172.21.15.84:0/2038543439' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:53:32.953 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:53:32.986 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:32 smithi084 conmon[33408]: cluster 2024-01-13T09:53:30.878591+0000 mgr.smithi084.itrhok (mgr.14180) 263 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:32.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:32 smithi102 conmon[37544]: cluster 2024-01-13T09:53:30.878591+0000 mgr.smithi084.itrhok (mgr.14180) 263 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:34.746 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:53:34.968 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:34 smithi084 conmon[33408]: cluster 2024-01-13T09:53:32.879013+0000 mgr.smithi084.itrhok (mgr 2024-01-13T09:53:34.969 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:34 smithi084 conmon[33408]: .14180) 264 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:34.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:34 smithi102 conmon[37544]: cluster 2024-01-13T09:53:32.879013+0000 mgr.smithi084.itrhok (mgr.14180) 264 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:35.362 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:53:35.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:35 smithi084 conmon[33408]: audit 2024-01-13T09:53:34.752198+0000 mon.smithi084 (mon.0) 587 : audit [DBG] from='client.? 172.21.15.84:0/1879525321' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:53:35.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:35 smithi102 conmon[37544]: audit 2024-01-13T09:53:34.752198+0000 mon.smithi084 (mon.0) 587 : audit [DBG] from='client.? 172.21.15.84:0/1879525321' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:53:36.363 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:53:36.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:36 smithi084 conmon[33408]: cluster 2024-01-13T09:53:34.879467+0000 2024-01-13T09:53:36.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:36 smithi084 conmon[33408]: mgr.smithi084.itrhok (mgr.14180) 265 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:36.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:36 smithi102 conmon[37544]: cluster 2024-01-13T09:53:34.879467+0000 mgr.smithi084.itrhok (mgr.14180) 265 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:37.889 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:37 smithi102 conmon[37544]: audit 2024-01-13T09:53:36.866939+0000 mon.smithi084 (mon. 2024-01-13T09:53:37.889 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:37 smithi102 conmon[37544]: 0) 588 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T09:53:37.889 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:37 smithi102 conmon[37544]: audit 2024-01-13T09:53:37.220587+0000 mon.smithi084 (mon.0) 589 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:53:37.890 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:37 smithi102 conmon[37544]: audit 2024-01-13T09:53:37.221181+0000 mon.smithi084 (mon.0) 590 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T09:53:37.890 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:37 smithi102 conmon[37544]: audit 2024-01-13T09:53:37.225070+0000 mon.smithi084 (mon.0) 591 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:53:37.890 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:37 smithi102 conmon[37544]: audit 2024-01-13T09:53:37.228472+0000 mon.smithi084 (mon.0) 592 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T09:53:37.890 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:37 smithi102 conmon[37544]: audit 2024-01-13T09:53:37.231034+0000 mon.smithi084 (mon.0) 593 2024-01-13T09:53:37.890 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:37 smithi102 conmon[37544]: : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:53:37.890 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:37 smithi102 conmon[37544]: audit 2024-01-13T09:53:37.231846+0000 mon.smithi084 (mon.0) 594 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:53:37.890 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:37 smithi102 conmon[37544]: audit 2024-01-13T09:53:37.232048 2024-01-13T09:53:37.890 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:37 smithi102 conmon[37544]: +0000 mon.smithi084 (mon.0) 595 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:53:37.890 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:37 smithi102 conmon[37544]: 2024-01-13T09:53:37.890 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:37 smithi102 conmon[37544]: audit 2024-01-13T09:53:37.890 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:37 smithi102 conmon[37544]: 2024-01-13T09:53:37.232502+0000 mon.smithi084 (mon.0) 2024-01-13T09:53:37.890 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:37 smithi102 conmon[37544]: 596 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:53:37.918 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:37 smithi084 conmon[33408]: audit 2024-01-13T09:53:36.866939+0000 mon.smithi084 (mon.0) 2024-01-13T09:53:37.918 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:37 smithi084 conmon[33408]: 588 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T09:53:37.918 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:37 smithi084 conmon[33408]: audit 2024-01-13T09:53:37.220587+0000 mon.smithi084 (mon.0) 589 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:53:37.918 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:37 smithi084 conmon[33408]: audit 2024-01-13T09:53:37.221181+0000 mon.smithi084 (mon.0) 590 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T09:53:37.918 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:37 smithi084 conmon[33408]: audit 2024-01-13T09:53:37.225070+0000 mon.smithi084 (mon.0) 591 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:53:37.918 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:37 smithi084 conmon[33408]: audit 2024-01-13T09:53:37.228472+0000 mon.smithi084 (mon.0) 592 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T09:53:37.918 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:37 smithi084 conmon[33408]: audit 2024-01-13T09:53:37.231034+0000 mon.smithi084 (mon.0) 593 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:53:37.919 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:37 smithi084 conmon[33408]: audit 2024-01-13T09:53:37.231846+0000 mon.smithi084 (mon.0) 594 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:53:37.919 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:37 smithi084 conmon[33408]: audit 2024-01-13T09:53:37.232048+0000 mon.smithi084 (mon.0) 595 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:53:37.919 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:37 smithi084 conmon[33408]: audit 2024-01-13T09:53:37.232502+0000 mon.smithi084 (mon.0) 596 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:53:38.343 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:53:38.746 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:38 smithi102 conmon[37544]: cluster 2024-01-13T09:53:36.879820+0000 mgr.smithi084.itrhok (mgr.14180) 266 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:38.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:38 smithi102 conmon[37544]: cluster 2024-01-13T09:53:37.225377+0000 mgr.smithi084.itrhok (mgr.14180) 267 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:38.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:38 smithi102 conmon[37544]: cluster 2024-01-13T09:53:37.578415+0000 mon.smithi084 (mon.0) 597 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T09:53:38.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:38 smithi102 conmon[37544]: cluster 2024-01-13T09:53:37.578500+0000 mon.smithi084 (mon.0) 598 : cluster [INF] Cluster is now healthy 2024-01-13T09:53:38.747 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:38 smithi102 conmon[37544]: audit 2024-01-13T09:53:38.348592+0000 mon.smithi084 (mon.0) 599 : audit [DBG] from='client.? 172.21.15.84:0/3818037627' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:53:38.878 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:38 smithi084 conmon[33408]: cluster 2024-01-13T09:53:36.879820+0000 mgr.smithi084.itrhok (mgr.14180) 266 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:38.878 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:38 smithi084 conmon[33408]: cluster 2024-01-13T09:53:37.225377+0000 mgr.smithi084.itrhok (mgr.14180) 267 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:38.878 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:38 smithi084 conmon[33408]: cluster 2024-01-13T09:53:37.578415+0000 mon.smithi084 (mon.0) 597 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T09:53:38.878 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:38 smithi084 conmon[33408]: cluster 2024-01-13T09:53:37.578500+0000 mon.smithi084 (mon.0) 598 : cluster [INF] Cluster is now healthy 2024-01-13T09:53:38.878 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:38 smithi084 conmon[33408]: audit 2024-01-13T09:53:38.348592+0000 mon.smithi084 (mon.0) 599 : audit [DBG] from='client.? 172.21.15.84:0/3818037627' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:53:39.673 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:53:40.675 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:53:40.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:40 smithi102 conmon[37544]: cluster 2024-01-13T09:53:39.225705+0000 mgr.smithi084.itrhok (mgr.14180) 268 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:40.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:40 smithi102 conmon[37544]: audit 2024-01-13T09:53:39.787165+0000 mon.smithi102 (mon.1) 14 : audit [INF] from='client.? 172.21.15.102:0/2622601594' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0aeb54c0-0568-4739-a267-7a64c81598e8"}]: dispatch 2024-01-13T09:53:40.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:40 smithi102 conmon[37544]: audit 2024-01-13T09:53:39.787538+0000 mon.smithi084 (mon.0) 600 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0aeb54c0-0568-4739-a267-7a64c81598e8"}]: dispatch 2024-01-13T09:53:40.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:40 smithi102 conmon[37544]: audit 2024-01-13T09:53:39.790786+0000 mon.smithi084 (mon.0) 601 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0aeb54c0-0568-4739-a267-7a64c81598e8"}]': finished 2024-01-13T09:53:40.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:40 smithi102 conmon[37544]: cluster 2024-01-13T09:53:39.790835+0000 mon.smithi084 (mon.0) 602 : cluster [DBG] osdmap e30: 1 total, 0 up, 1 in 2024-01-13T09:53:40.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:40 smithi102 conmon[37544]: audit 2024-01-13T09:53:39.790931+0000 mon.smithi084 (mon.0) 603 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:53:40.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:40 smithi102 conmon[37544]: audit 2024-01-13T09:53:40.076768+0000 mon.smithi102 ( 2024-01-13T09:53:40.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:40 smithi102 conmon[37544]: mon.1) 15 : audit [INF] from='client.? 172.21.15.102:0/3336525419' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:53:40.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:40 smithi102 conmon[37544]: audit 2024-01-13T09:53:40.077341+0000 mon.smithi084 (mon.0) 604 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:53:40.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:40 smithi102 conmon[37544]: audit 2024-01-13T09:53:40.080719+0000 mon.smithi084 (mon.0) 605 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:53:40.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:40 smithi102 conmon[37544]: cluster 2024-01-13T09:53:40.080765+0000 mon.smithi084 (mon.0) 606 : cluster [DBG] osdmap e31: 0 total, 0 up, 0 in 2024-01-13T09:53:41.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:40 smithi084 conmon[33408]: cluster 2024-01-13T09:53:39.225705+0000 mgr.smithi084.itrhok (mgr.14180) 268 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:41.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:40 smithi084 conmon[33408]: audit 2024-01-13T09:53:39.787165+0000 mon.smithi102 (mon.1) 14 : audit [INF] from='client.? 172.21.15.102:0/2622601594' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0aeb54c0-0568-4739-a267-7a64c81598e8"}]: dispatch 2024-01-13T09:53:41.031 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:40 smithi084 conmon[33408]: audit 2024-01-13T09:53:39.787538+0000 mon.smithi084 (mon.0) 600 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0aeb54c0-0568-4739-a267-7a64c81598e8"}]: dispatch 2024-01-13T09:53:41.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:40 smithi084 conmon[33408]: audit 2024-01-13T09:53:39.790786+0000 mon.smithi084 (mon.0) 601 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0aeb54c0-0568-4739-a267-7a64c81598e8"}]': finished 2024-01-13T09:53:41.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:40 smithi084 conmon[33408]: cluster 2024-01-13T09:53:39.790835+0000 mon.smithi084 (mon.0) 602 : cluster [DBG] osdmap e30: 1 total, 0 up, 1 in 2024-01-13T09:53:41.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:40 smithi084 conmon[33408]: audit 2024-01-13T09:53:39.790931+0000 mon.smithi084 (mon.0) 603 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:53:41.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:40 smithi084 conmon[33408]: audit 2024-01-13T09:53:40.076768+0000 mon.smithi102 (mon.1) 15 : audit [INF] from='client.? 172.21.15.102:0/3336525419' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:53:41.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:40 smithi084 conmon[33408]: audit 2024-01-13T09:53:40.077341+0000 mon.smithi084 (mon.0) 604 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:53:41.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:40 smithi084 conmon[33408]: audit 2024-01-13T09:53:40.080719+0000 mon.smithi084 (mon.0) 605 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:53:41.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:40 smithi084 conmon[33408]: cluster 2024-01-13T09:53:40.080765+0000 mon.smithi084 (mon.0) 606 : cluster [DBG] osdmap e31: 0 total, 0 up, 0 in 2024-01-13T09:53:41.846 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:41 smithi084 conmon[33408]: audit 2024-01-13T09:53:40.735336+0000 mon.smithi084 (mon.0 2024-01-13T09:53:41.846 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:41 smithi084 conmon[33408]: ) 607 : audit [INF] from='client.? 172.21.15.84:0/1849138853' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "545b66d0-0316-4c98-a69c-16efba2a0a69"}]: dispatch 2024-01-13T09:53:41.847 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:41 smithi084 conmon[33408]: audit 2024-01-13T09:53:40.743946+0000 mon.smithi084 (mon.0) 608 : audit [INF] from='client.? 172.21.15.84:0/1849138853' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "545b66d0-0316-4c98-a69c-16efba2a0a69"}]': finished 2024-01-13T09:53:41.847 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:41 smithi084 conmon[33408]: cluster 2024-01-13T09:53:40.744044+0000 mon.smithi084 (mon.0) 609 : cluster [DBG] osdmap e32: 1 total, 0 up, 1 in 2024-01-13T09:53:41.847 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:41 smithi084 conmon[33408]: audit 2024-01-13T09:53:40.744428+0000 mon.smithi084 (mon.0) 610 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:53:41.847 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:41 smithi084 conmon[33408]: audit 2024-01-13T09:53:41.087704+0000 mon.smithi084 (mon.0) 611 : audit [INF] from='client.? 172.21.15.84:0/75266653' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:53:41.847 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:41 smithi084 conmon[33408]: audit 2024-01-13T09:53:41.093656+0000 mon.smithi084 (mon.0) 612 : audit [INF] from='client.? 172.21.15.84:0/75266653' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:53:41.847 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:41 smithi084 conmon[33408]: cluster 2024-01-13T09:53:41.093695+0000 mon.smithi084 (mon.0) 613 : cluster [DBG] osdmap e33: 0 total, 0 up, 0 in 2024-01-13T09:53:41.988 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:41 smithi102 conmon[37544]: audit 2024-01-13T09:53:40.735336+0000 mon.smithi084 (mon.0) 607 : audit [INF] from='client.? 172.21.15.84:0/1849138853' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "545b66d0-0316-4c98-a69c-16efba2a0a69"}]: dispatch 2024-01-13T09:53:41.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:41 smithi102 conmon[37544]: audit 2024-01-13T09:53:40.743946+0000 mon.smithi084 (mon.0) 608 : audit [INF] from='client.? 172.21.15.84:0/1849138853' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "545b66d0-0316-4c98-a69c-16efba2a0a69"}]': finished 2024-01-13T09:53:41.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:41 smithi102 conmon[37544]: cluster 2024-01-13T09:53:40.744044+0000 mon.smithi084 (mon.0) 609 : cluster [DBG] osdmap e32: 1 total, 0 up, 1 in 2024-01-13T09:53:41.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:41 smithi102 conmon[37544]: audit 2024-01 2024-01-13T09:53:41.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:41 smithi102 conmon[37544]: -13T09:53:40.744428+0000 mon.smithi084 (mon.0) 610 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:53:41.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:41 smithi102 conmon[37544]: audit 2024-01-13T09:53:41.087704 2024-01-13T09:53:41.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:41 smithi102 conmon[37544]: +0000 mon.smithi084 (mon.0) 611 : audit [INF] from='client.? 172.21.15.84:0/75266653' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:53:41.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:41 smithi102 conmon[37544]: 2024-01-13T09:53:41.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:41 smithi102 conmon[37544]: audit 2024-01 2024-01-13T09:53:41.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:41 smithi102 conmon[37544]: -13T09:53:41.093656+0000 mon.smithi084 (mon.0) 612 : 2024-01-13T09:53:41.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:41 smithi102 conmon[37544]: audit [INF] from='client.? 172.21.15.84:0/75266653' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:53:41.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:41 smithi102 conmon[37544]: cluster 2024-01-13T09:53:41.093695 2024-01-13T09:53:41.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:41 smithi102 conmon[37544]: +0000 mon.smithi084 (mon.0) 613 : cluster [DBG] osdmap e33: 0 total, 0 up, 0 in 2024-01-13T09:53:41.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:41 smithi102 conmon[37544]: 2024-01-13T09:53:42.853 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: cephadm 2024-01-13T09:53:40.787328+0000 mgr.smithi084.itrhok (mgr.14180) 269 : 2024-01-13T09:53:42.853 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: cephadm [ERR] executing create_from_spec_one(([('smithi084', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi102', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:53:42.853 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:53:42.853 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:53:42.853 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: return f(*arg) 2024-01-13T09:53:42.853 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:53:42.854 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:53:42.854 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:53:42.854 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: code, '\n'.join(err))) 2024-01-13T09:53:42.854 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:53:42.854 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmppc2_izex:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmplz93pyef:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:53:42.854 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:53:42.854 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:53:42.854 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:53:42.854 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 0aeb54c0-0568-4739-a267-7a64c81598e8 2024-01-13T09:53:42.854 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:53:42.854 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:53:42.854 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:53:42.854 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:53:42.854 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:53:42.855 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr self.prepare() 2024-01-13T09:53:42.855 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:53:42.855 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:53:42.855 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:53:42.855 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:53:42.855 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:53:42.855 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:53:42.855 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:53:42.855 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:53:42.855 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:53:42.855 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:53:42.855 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:53:42.855 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:53:42.855 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:53:42.855 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:53:42.856 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:53:42.856 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:53:42.856 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:53:42.856 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:53:42.856 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:53:42.856 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:53:42.856 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:53:42.856 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:53:42.856 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:53:42.856 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:53:42.856 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:53:42.856 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:53:42.856 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:53:42.857 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:53:42.857 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:53:42.857 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:53:42.857 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:53:42.857 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:53:42.857 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:53:42.857 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:53:42.857 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:53:42.857 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:53:42.857 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:53:42.857 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:53:42.857 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:53:42.857 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:53:42.857 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:53:42.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:53:42.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:53:42.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:53:42.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:53:42.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:53:42.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:53:42.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:53:42.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:53:42.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:53:42.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:53:42.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:53:42.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: main() 2024-01-13T09:53:42.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:53:42.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: r = ctx.func(ctx) 2024-01-13T09:53:42.859 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:53:42.859 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:53:42.859 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:53:42.859 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:53:42.859 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:53:42.859 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:53:42.859 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90 2024-01-13T09:53:42.859 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: d2", line 1927, in _validate_fsid 2024-01-13T09:53:42.859 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:53:42.859 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:53:42.859 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:53:42.859 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:53:42.859 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:53:42.860 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmppc2_izex:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmplz93pyef:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:53:42.860 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: cluster 2024-01-13T09:53:41.226000+0000 mgr.smithi084.itrhok (mgr.14180) 270 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:42.860 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:42 smithi084 conmon[33408]: cluster 2024-01-13T09:53:42.089561+0000 mon.smithi084 (mon.0) 614 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T09:53:42.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: cephadm 2024-01-13T09:53:40.787328+0000 mgr.smithi084.itrhok (mgr.14180) 269 : cephadm [ERR] executing create_from_spec_one(([('smithi084', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi102', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:53:42.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:53:42.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:53:42.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: return f(*arg) 2024-01-13T09:53:42.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:53:42.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:53:42.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:53:42.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: code, '\n'.join(err))) 2024-01-13T09:53:42.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:53:42.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmppc2_izex:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmplz93pyef:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:53:42.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:53:42.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:53:42.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:53:42.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 0aeb54c0-0568-4739-a267-7a64c81598e8 2024-01-13T09:53:42.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:53:42.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:53:42.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:53:42.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:53:42.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:53:42.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr self.prepare() 2024-01-13T09:53:42.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:53:42.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:53:42.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:53:42.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:53:42.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:53:42.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:53:42.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:53:42.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:53:42.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:53:42.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:53:42.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:53:42.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:53:42.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:53:42.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:53:42.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:53:42.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:53:42.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:53:42.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:53:42.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:53:42.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:53:42.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:53:42.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:53:42.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:53:42.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:53:42.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:53:42.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:53:42.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:53:42.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:53:42.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:53:42.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:53:42.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:53:42.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:53:42.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:53:42.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:53:42.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:53:42.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:53:42.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:53:42.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:53:42.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:53:42.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:53:42.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:53:42.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:53:42.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:53:42.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:53:42.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:53:42.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:53:42.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:53:42.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:53:42.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:53:42.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:53:42.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:53:42.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:53:42.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: main() 2024-01-13T09:53:42.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:53:42.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: r = ctx.func(ctx) 2024-01-13T09:53:42.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:53:42.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:53:42.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:53:42.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:53:42.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:53:42.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:53:42.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-8777 2024-01-13T09:53:42.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: 4f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:53:42.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:53:42.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:53:42.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:53:42.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:53:42.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:53:42.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmppc2_izex:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmplz93pyef:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:53:42.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: cluster 2024-01-13T09:53:41.226000+0000 mgr.smithi084.itrhok (mgr.14180) 270 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:42.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:42 smithi102 conmon[37544]: cluster 2024-01-13T09:53:42.089561+0000 mon.smithi084 (mon.0) 614 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T09:53:43.210 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:53:43.857 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: cephadm 2024-01-13T09:53:41.852285+0000 mgr.smithi084.itrhok (mgr.14180) 271 : cephadm [ERR] executing create_from_spec_one(([('smithi084', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi102', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:53:43.857 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:53:43.857 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:53:43.857 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: return f(*arg) 2024-01-13T09:53:43.857 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:53:43.857 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:53:43.857 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:53:43.857 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: code, '\n'.join(err))) 2024-01-13T09:53:43.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi084/config 2024-01-13T09:53:43.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp_hila3uk:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp21p7n4qw:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:53:43.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:53:43.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:53:43.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:53:43.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 545b66d0-0316-4c98-a69c-16efba2a0a69 2024-01-13T09:53:43.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:53:43.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:53:43.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:53:43.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:53:43.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:53:43.858 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr self.prepare() 2024-01-13T09:53:43.859 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:53:43.859 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:53:43.859 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:53:43.859 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:53:43.859 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:53:43.859 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:53:43.859 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:53:43.859 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:53:43.859 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:53:43.859 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:53:43.859 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:53:43.860 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:53:43.860 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:53:43.860 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:53:43.860 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:53:43.860 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:53:43.860 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:53:43.860 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:53:43.860 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:53:43.860 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:53:43.860 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:53:43.860 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:53:43.860 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:53:43.860 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:53:43.860 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:53:43.861 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:53:43.861 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:53:43.861 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:53:43.861 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:53:43.861 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:53:43.861 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:53:43.861 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:53:43.861 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:53:43.861 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:53:43.861 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:53:43.861 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:53:43.861 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:53:43.861 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:53:43.861 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:53:43.862 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:53:43.862 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:53:43.862 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:53:43.862 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:53:43.862 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:53:43.862 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:53:43.862 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:53:43.862 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:53:43.862 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:53:43.862 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:53:43.862 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:53:43.862 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:53:43.862 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:53:43.862 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: main() 2024-01-13T09:53:43.862 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:53:43.863 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: r = ctx.func(ctx) 2024-01-13T09:53:43.863 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:53:43.863 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:53:43.863 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:53:43.863 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:53:43.863 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:53:43.863 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:53:43.863 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-8777 2024-01-13T09:53:43.863 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: 4f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:53:43.863 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:53:43.863 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:53:43.863 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:53:43.863 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:53:43.864 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:53:43.865 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:53:43.865 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp_hila3uk:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp21p7n4qw:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:53:43.866 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: cephadm 2024-01-13T09:53:41.853856+0000 mgr.smithi084.itrhok (mgr.14180) 272 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T09:53:43.866 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: service_id: all-available-devices 2024-01-13T09:53:43.866 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: service_name: osd.all-available-devices 2024-01-13T09:53:43.866 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: placement: 2024-01-13T09:53:43.866 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: host_pattern: '*' 2024-01-13T09:53:43.866 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: spec: 2024-01-13T09:53:43.866 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: data_devices: 2024-01-13T09:53:43.866 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: all: true 2024-01-13T09:53:43.866 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: filter_logic: AND 2024-01-13T09:53:43.866 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: objectstore: bluestore 2024-01-13T09:53:43.866 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:53:43.866 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmppc2_izex:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmplz93pyef:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:53:43.866 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:53:43.867 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:53:43.867 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:53:43.867 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 0aeb54c0-0568-4739-a267-7a64c81598e8 2024-01-13T09:53:43.867 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:53:43.867 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:53:43.867 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:53:43.867 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:53:43.867 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:53:43.867 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr self.prepare() 2024-01-13T09:53:43.867 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:53:43.867 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:53:43.867 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:53:43.867 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:53:43.867 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:53:43.867 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:53:43.867 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:53:43.867 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:53:43.868 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:53:43.868 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:53:43.868 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:53:43.868 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:53:43.868 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:53:43.868 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:53:43.868 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:53:43.868 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:53:43.868 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:53:43.868 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:53:43.868 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:53:43.868 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:53:43.868 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:53:43.868 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:53:43.868 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:53:43.869 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:53:43.869 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:53:43.869 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:53:43.869 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:53:43.869 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:53:43.869 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:53:43.869 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:53:43.869 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:53:43.869 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:53:43.869 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:53:43.869 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:53:43.869 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:53:43.869 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:53:43.869 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:53:43.870 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:53:43.870 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:53:43.870 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:53:43.870 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:53:43.870 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:53:43.870 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:53:43.870 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:53:43.870 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:53:43.870 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/ 2024-01-13T09:53:43.870 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: lvm/zap.py", line 302, in zap_osd 2024-01-13T09:53:43.870 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:53:43.870 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:53:43.870 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:53:43.870 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:53:43.870 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:53:43.870 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:53:43.871 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: main() 2024-01-13T09:53:43.871 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:53:43.871 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: r = ctx.func(ctx) 2024-01-13T09:53:43.871 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:53:43.871 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:53:43.871 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:53:43.871 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:53:43.871 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:53:43.871 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:53:43.871 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:53:43.871 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:53:43.871 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:53:43.871 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:53:43.871 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:53:43.871 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:53:43.871 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmppc2_izex:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmplz93pyef:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:53:43.872 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:53:43.872 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T09:53:43.872 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: if self._apply_service(spec): 2024-01-13T09:53:43.872 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T09:53:43.872 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T09:53:43.872 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T09:53:43.872 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T09:53:43.872 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T09:53:43.872 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T09:53:43.872 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T09:53:43.872 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T09:53:43.872 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T09:53:43.872 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: raise self._value 2024-01-13T09:53:43.872 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T09:53:43.873 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: result = (True, func(*args, **kwds)) 2024-01-13T09:53:43.873 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T09:53:43.873 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: return list(map(*args)) 2024-01-13T09:53:43.873 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:53:43.873 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: return f(*arg) 2024-01-13T09:53:43.873 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:53:43.873 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:53:43.873 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:53:43.873 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: code, '\n'.join(err))) 2024-01-13T09:53:43.873 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:53:43.873 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmppc2_izex:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmplz93pyef:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:53:43.873 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:53:43.873 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:53:43.873 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:53:43.873 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 0aeb54c0-0568-4739-a267-7a64c81598e8 2024-01-13T09:53:43.873 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:53:43.874 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:53:43.874 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:53:43.874 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:53:43.874 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:53:43.874 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr self.prepare() 2024-01-13T09:53:43.874 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:53:43.874 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:53:43.874 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:53:43.874 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:53:43.874 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:53:43.874 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:53:43.874 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:53:43.874 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:53:43.874 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:53:43.874 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:53:43.874 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:53:43.874 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:53:43.875 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:53:43.875 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:53:43.875 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:53:43.875 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:53:43.875 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:53:43.875 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:53:43.875 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:53:43.875 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:53:43.875 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:53:43.875 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:53:43.875 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:53:43.875 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:53:43.875 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:53:43.875 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:53:43.875 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:53:43.875 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:53:43.876 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:53:43.876 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:53:43.876 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:53:43.876 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:53:43.876 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:53:43.876 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:53:43.876 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:53:43.876 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:53:43.876 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:53:43.876 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:53:43.876 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:53:43.876 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:53:43.876 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:53:43.876 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:53:43.877 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:53:43.877 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:53:43.877 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:53:43.877 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:53:43.877 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:53:43.877 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:53:43.877 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:53:43.877 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:53:43.877 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:53:43.877 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:53:43.877 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:53:43.877 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: main() 2024-01-13T09:53:43.877 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:53:43.877 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: r = ctx.func(ctx) 2024-01-13T09:53:43.877 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:53:43.878 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:53:43.878 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:53:43.878 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:53:43.878 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:53:43.878 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:53:43.878 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:53:43.878 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:53:43.878 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:53:43.878 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:53:43.878 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:53:43.878 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:53:43.878 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmppc2_izex:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmplz93pyef:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:53:43.878 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: cluster 2024-01-13T09:53:41.854461+0000 mgr.smithi084.itrhok (mgr.14180) 273 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:43.878 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:43 smithi084 conmon[33408]: audit 2024-01-13T09:53:43.215696+0000 mon.smithi084 (mon.0) 615 : audit [DBG] from='client.? 172.21.15.84:0/1555912450' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:53:43.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: cephadm 2024-01-13T09:53:41.852285+0000 mgr.smithi084.itrhok (mgr.14180) 271 : cephadm [ERR] executing create_from_spec_one(([('smithi084', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi102', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:53:43.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:53:43.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:53:43.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: return f(*arg) 2024-01-13T09:53:43.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:53:43.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:53:43.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:53:43.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: code, '\n'.join(err))) 2024-01-13T09:53:43.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi084/config 2024-01-13T09:53:43.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp_hila3uk:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp21p7n4qw:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:53:43.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:53:43.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:53:43.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:53:43.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 545b66d0-0316-4c98-a69c-16efba2a0a69 2024-01-13T09:53:43.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:53:43.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:53:43.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:53:43.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:53:43.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:53:43.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr self.prepare() 2024-01-13T09:53:43.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:53:43.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:53:43.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:53:43.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:53:43.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:53:43.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:53:43.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:53:43.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:53:43.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:53:43.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:53:43.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:53:43.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:53:43.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:53:43.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:53:43.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:53:43.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:53:43.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:53:43.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:53:43.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:53:43.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:53:43.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:53:43.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:53:43.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:53:43.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:53:43.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:53:43.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:53:43.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:53:43.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:53:43.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:53:43.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:53:43.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:53:43.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:53:43.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:53:43.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:53:43.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:53:43.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:53:43.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:53:43.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:53:43.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:53:43.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:53:43.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:53:43.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:53:43.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:53:43.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:53:43.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:53:43.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:53:43.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:53:43.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:53:43.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:53:43.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:53:43.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:53:43.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:53:43.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: main() 2024-01-13T09:53:43.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:53:43.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: r = ctx.func(ctx) 2024-01-13T09:53:43.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:53:43.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:53:43.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:53:43.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:53:43.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:53:43.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:53:43.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-8777 2024-01-13T09:53:43.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: 4f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:53:43.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:53:43.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:53:43.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:53:43.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:53:43.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:53:43.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp_hila3uk:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp21p7n4qw:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:53:43.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: cephadm 2024-01-13T09:53:41.853856+0000 mgr.smithi084.itrhok (mgr.14180) 272 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T09:53:43.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: service_id: all-available-devices 2024-01-13T09:53:43.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: service_name: osd.all-available-devices 2024-01-13T09:53:43.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: placement: 2024-01-13T09:53:43.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: host_pattern: '*' 2024-01-13T09:53:43.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: spec: 2024-01-13T09:53:43.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: data_devices: 2024-01-13T09:53:43.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: all: true 2024-01-13T09:53:43.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: filter_logic: AND 2024-01-13T09:53:43.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: objectstore: bluestore 2024-01-13T09:53:43.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:53:43.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmppc2_izex:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmplz93pyef:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:53:43.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:53:43.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:53:43.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:53:43.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 0aeb54c0-0568-4739-a267-7a64c81598e8 2024-01-13T09:53:43.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:53:43.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:53:43.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:53:43.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:53:43.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:53:43.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr self.prepare() 2024-01-13T09:53:43.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:53:43.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:53:43.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:53:43.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:53:43.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:53:43.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:53:43.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:53:43.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:53:43.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:53:43.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:53:43.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:53:43.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:53:43.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:53:43.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:53:43.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:53:43.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:53:43.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:53:43.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:53:43.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:53:43.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:53:43.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:53:43.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:53:43.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:53:43.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:53:43.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:53:43.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:53:43.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:53:43.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:53:43.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:53:43.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:53:43.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:53:43.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:53:44.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:53:44.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:53:44.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:53:44.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:53:44.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:53:44.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:53:44.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:53:44.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:53:44.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:53:44.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:53:44.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:53:44.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:53:44.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:53:44.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/ 2024-01-13T09:53:44.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: lvm/zap.py", line 302, in zap_osd 2024-01-13T09:53:44.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:53:44.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:53:44.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:53:44.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:53:44.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:53:44.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:53:44.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: main() 2024-01-13T09:53:44.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:53:44.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: r = ctx.func(ctx) 2024-01-13T09:53:44.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:53:44.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:53:44.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:53:44.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:53:44.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:53:44.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:53:44.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:53:44.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:53:44.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:53:44.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:53:44.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:53:44.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:53:44.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmppc2_izex:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmplz93pyef:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:53:44.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:53:44.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T09:53:44.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: if self._apply_service(spec): 2024-01-13T09:53:44.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T09:53:44.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T09:53:44.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T09:53:44.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T09:53:44.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T09:53:44.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T09:53:44.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T09:53:44.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T09:53:44.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T09:53:44.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: raise self._value 2024-01-13T09:53:44.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T09:53:44.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: result = (True, func(*args, **kwds)) 2024-01-13T09:53:44.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T09:53:44.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: return list(map(*args)) 2024-01-13T09:53:44.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:53:44.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: return f(*arg) 2024-01-13T09:53:44.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:53:44.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:53:44.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:53:44.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: code, '\n'.join(err))) 2024-01-13T09:53:44.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:53:44.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmppc2_izex:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmplz93pyef:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:53:44.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:53:44.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:53:44.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:53:44.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 0aeb54c0-0568-4739-a267-7a64c81598e8 2024-01-13T09:53:44.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:53:44.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:53:44.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:53:44.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:53:44.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:53:44.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr self.prepare() 2024-01-13T09:53:44.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:53:44.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:53:44.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:53:44.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:53:44.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:53:44.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:53:44.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:53:44.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:53:44.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:53:44.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:53:44.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:53:44.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:53:44.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:53:44.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:53:44.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:53:44.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:53:44.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:53:44.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:53:44.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:53:44.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:53:44.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:53:44.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:53:44.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:53:44.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:53:44.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:53:44.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:53:44.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:53:44.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:53:44.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:53:44.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:53:44.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:53:44.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:53:44.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:53:44.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:53:44.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:53:44.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:53:44.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:53:44.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:53:44.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:53:44.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:53:44.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:53:44.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:53:44.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:53:44.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:53:44.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:53:44.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:53:44.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:53:44.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:53:44.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:53:44.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:53:44.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:53:44.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:53:44.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:53:44.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: main() 2024-01-13T09:53:44.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:53:44.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: r = ctx.func(ctx) 2024-01-13T09:53:44.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:53:44.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:53:44.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:53:44.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:53:44.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:53:44.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:53:44.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:53:44.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:53:44.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:53:44.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:53:44.009 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:53:44.009 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:53:44.009 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmppc2_izex:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmplz93pyef:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:53:44.009 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: cluster 2024-01-13T09:53:41.854461+0000 mgr.smithi084.itrhok (mgr.14180) 273 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:44.009 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:43 smithi102 conmon[37544]: audit 2024-01-13T09:53:43.215696+0000 mon.smithi084 (mon.0) 615 : audit [DBG] from='client.? 172.21.15.84:0/1555912450' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:53:44.865 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:53:45.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:45 smithi102 conmon[37544]: cluster 2024-01-13T09:53:43.854842+0000 mgr.smithi084.itrhok (mgr.14180) 274 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:46.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:45 smithi084 conmon[33408]: cluster 2024-01-13T09:53:43.854842+0000 mgr.smithi084.itrhok ( 2024-01-13T09:53:46.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:45 smithi084 conmon[33408]: mgr.14180) 274 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:46.855 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:53:47.523 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:53:47.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:47 smithi084 conmon[33408]: cluster 2024-01-13T09:53:45.855273+0000 mgr.smithi084.itrhok (mgr.14180) 275 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:47.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:47 smithi084 conmon[33408]: audit 2024-01-13T09:53:46.860615+0000 mon.smithi084 (mon.0) 616 : audit [DBG] from='client.? 172.21.15.84:0/4178851293' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:53:47.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:47 smithi102 conmon[37544]: cluster 2024-01-13T09:53:45.855273+0000 mgr.smithi084.itrhok (mgr.14180) 275 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:47.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:47 smithi102 conmon[37544]: audit 2024-01-13T09:53:46.860615+0000 2024-01-13T09:53:47.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:47 smithi102 conmon[37544]: mon.smithi084 (mon.0) 616 : audit [DBG] from='client.? 172.21.15.84:0/4178851293' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:53:48.525 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:53:49.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:49 smithi084 conmon[33408]: cluster 2024-01-13T09:53:47.855700+0000 mgr.smithi084.itrhok (mgr.14180) 276 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:49.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:49 smithi102 conmon[37544]: cluster 2024-01-13T09:53:47.855700+0000 mgr.smithi084.itrhok (mgr.14180) 276 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:50.532 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:53:51.164 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:53:51.531 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:51 smithi084 conmon[33408]: cluster 2024-01-13T09:53:49.856101+0000 mgr.smithi084.itrhok (mgr.14180 2024-01-13T09:53:51.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:51 smithi084 conmon[33408]: ) 277 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:51.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:51 smithi084 conmon[33408]: audit 2024-01-13T09:53:50.537069+0000 mon.smithi084 (mon.0) 617 : audit [DBG] from='client.? 172.21.15.84:0/2821139526' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:53:51.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:51 smithi102 conmon[37544]: cluster 2024-01-13T09:53:49.856101+0000 mgr.smithi084.itrhok (mgr.14180) 277 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:51.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:51 smithi102 conmon[37544]: audit 2024-01-13 2024-01-13T09:53:51.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:51 smithi102 conmon[37544]: T09:53:50.537069+0000 mon.smithi084 (mon.0) 617 : audit [DBG] from='client.? 172.21.15.84:0/2821139526' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:53:52.165 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:53:53.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:53 smithi102 conmon[37544]: cluster 2024-01-13T09:53:51.856549+0000 mgr.smithi084.itrhok (mgr.14180) 278 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:53.750 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:53 smithi084 conmon[33408]: cluster 2024-01-13T09:53:51.856549+0000 mgr.smithi084.itrhok (mgr.14180 2024-01-13T09:53:53.750 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:53 smithi084 conmon[33408]: ) 278 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:54.112 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:53:54.531 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:54 smithi084 conmon[33408]: audit 2024-01-13T09:53:54.117114+0000 mon.smithi084 (mon.0) 618 : 2024-01-13T09:53:54.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:54 smithi084 conmon[33408]: audit [DBG] from='client.? 172.21.15.84:0/368557113' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:53:54.738 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:54 smithi102 conmon[37544]: audit 2024-01-13T09:53:54.117114+0000 mon.smithi084 (mon.0) 618 : audit [DBG] from='client.? 172.21.15.84:0/368557113' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:53:54.789 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:53:55.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:55 smithi102 conmon[37544]: cluster 2024-01-13T09:53:53.856926+0000 mgr.smithi084.itrhok (mgr.14180) 279 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:55.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:55 smithi084 conmon[33408]: cluster 2024-01-13T09:53:53.856926+0000 mgr.smithi084.itrhok (mgr.14180) 279 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:55.790 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:53:57.507 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:57 smithi084 conmon[33408]: cluster 2024-01-13T09:53:55.857346+0000 mgr.smithi084.itrhok (mgr.14180) 280 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:57.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:57 smithi102 conmon[37544]: cluster 2024-01-13T09:53:55.857346+0000 mgr.smithi084.itrhok (mgr.14180) 280 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:57.905 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:53:58.521 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:53:58.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:58 smithi102 conmon[37544]: audit 2024-01-13T09:53:57.910366+0000 mon.smithi084 (mon.0) 619 : audit [DBG] from='client.? 172.21.15.84:0/964319951' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:53:58.785 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:58 smithi084 conmon[33408]: audit 2024-01-13T09:53:57.910366+0000 mon.smithi084 (mon.0) 619 : audit [DBG] from='client.? 172.21.15.84:0/964319951' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:53:58.785 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:58 smithi084 conmon[33408]: 2024-01-13T09:53:59.522 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:53:59.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:53:59 smithi102 conmon[37544]: cluster 2024-01-13T09:53:57.857744+0000 mgr.smithi084.itrhok (mgr.14180) 281 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:53:59.781 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:53:59 smithi084 conmon[33408]: cluster 2024-01-13T09:53:57.857744+0000 mgr.smithi084.itrhok (mgr.14180) 281 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:01.461 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:54:01.713 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:01 smithi084 conmon[33408]: cluster 2024-01-13T09:53:59.858273+0000 mgr.smithi084.itrhok (mgr.14180) 282 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:01.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:01 smithi102 conmon[37544]: cluster 2024-01-13T09:53:59.858273+0000 mgr.smithi084.itrhok (mgr.14180) 282 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:02.096 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:54:02.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:02 smithi102 conmon[37544]: audit 2024-01-13T09:54:01.465330+0000 mon.smithi084 (mon.0) 620 : audit [DBG] from='client.? 172.21.15.84:0/1734279568' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:54:02.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:02 smithi084 conmon[33408]: audit 2024-01-13T09:54:01.465330+0000 mon.smithi084 (mon.0) 620 : audit 2024-01-13T09:54:02.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:02 smithi084 conmon[33408]: [DBG] from='client.? 172.21.15.84:0/1734279568' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:54:03.098 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:54:03.531 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:03 smithi084 conmon[33408]: cluster 2024-01-13T09:54:01.858637+0000 mgr.smithi084.itrhok (mgr.14180) 283 2024-01-13T09:54:03.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:03 smithi084 conmon[33408]: : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:03.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:03 smithi102 conmon[37544]: cluster 2024-01-13T09:54:01.858637+0000 mgr.smithi084.itrhok (mgr.14180) 283 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:05.079 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:54:05.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:05 smithi102 conmon[37544]: cluster 2024-01-13T09:54:03.859020+0000 mgr.smithi084.itrhok (mgr.14180) 284 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:05.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:05 smithi102 conmon[37544]: audit 2024-01-13T09:54:05.083417+0000 mon.smithi084 (mon.0) 621 : audit [DBG] from='client.? 172.21.15.84:0/2098981589' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:54:05.747 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:05 smithi084 conmon[33408]: cluster 2024-01-13T09:54:03.859020+0000 mgr.smithi084.itrhok (mgr.14180) 284 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:05.747 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:05 smithi084 conmon[33408]: audit 2024-01-13T09:54:05.083417+0000 mon.smithi084 (mon.0) 621 : audit [DBG] from='client.? 172.21.15.84:0/2098981589' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:54:05.766 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:54:06.767 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:54:07.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:07 smithi084 conmon[33408]: cluster 2024-01-13T09:54:05.859585+0000 mgr.smithi084.itrhok (mgr.14180) 285 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:07.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:07 smithi084 conmon[33408]: 2024-01-13T09:54:07.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:07 smithi102 conmon[37544]: cluster 2024-01-13T09:54:05.859585+0000 mgr.smithi084.itrhok (mgr.14180) 285 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:08.672 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:54:09.369 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:54:09.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:09 smithi102 conmon[37544]: cluster 2024-01-13T09:54:07.859890+0000 mgr.smithi084.itrhok (mgr.14180) 286 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:09.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:09 smithi102 conmon[37544]: audit 2024-01-13T09:54:08.676562+0000 mon.smithi084 (mon.0) 622 : audit [DBG] from='client.? 172.21.15.84:0/60187218' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:54:09.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:09 smithi084 conmon[33408]: cluster 2024-01-13T09:54:07.859890+0000 mgr.smithi084.itrhok (mgr.14180) 286 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:09.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:09 smithi084 conmon[33408]: audit 2024-01-13T09:54:08.676562+0000 mon.smithi084 (mon.0) 622 : audit [DBG] from='client.? 172.21.15.84:0/60187218' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:54:10.371 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:54:11.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:11 smithi084 conmon[33408]: cluster 2024-01-13T09:54:09.860288+0000 2024-01-13T09:54:11.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:11 smithi084 conmon[33408]: mgr.smithi084.itrhok (mgr.14180) 287 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:11.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:11 smithi102 conmon[37544]: cluster 2024-01-13T09:54:09.860288+0000 mgr.smithi084.itrhok (mgr.14180) 287 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:12.172 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:54:12.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:12 smithi084 conmon[33408]: audit 2024-01-13T09:54:12.176617+0000 mon.smithi084 (mon.0) 623 : audit [DBG] from='client.? 172.21.15.84:0/3750449983' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:54:12.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:12 smithi102 conmon[37544]: audit 2024-01-13T09:54:12.176617+0000 mon.smithi084 (mon.0) 623 : audit [DBG] from='client.? 172.21.15.84:0/3750449983' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:54:12.801 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:54:13.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:13 smithi102 conmon[37544]: cluster 2024-01-13T09:54:11.860748+0000 mgr.smithi084.itrhok (mgr.14180) 288 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:13.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:13 smithi084 conmon[33408]: cluster 2024-01-13T09:54:11.860748+0000 mgr.smithi084.itrhok (mgr.14180) 288 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:13.802 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:54:15.739 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:54:15.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:15 smithi102 conmon[37544]: cluster 2024-01-13T09:54:13.861141+0000 mgr.smithi084.itrhok (mgr.14180) 289 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:15.764 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:15 smithi084 conmon[33408]: cluster 2024-01-13T09:54:13.861141+0000 mgr.smithi084.itrhok (mgr.14180) 289 : cluster [DBG] 2024-01-13T09:54:15.764 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:15 smithi084 conmon[33408]: pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:16.334 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:54:16.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:16 smithi102 conmon[37544]: audit 2024-01-13T09:54:15.743890+0000 mon.smithi084 (mon.0) 624 : audit [DBG] from='client.? 172.21.15.84:0/1250048309' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:54:16.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:16 smithi084 conmon[33408]: audit 2024-01-13T09:54:15.743890+0000 mon.smithi084 (mon.0) 624 : audit [DBG] from='client.? 172.21.15.84:0/1250048309' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:54:17.335 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:54:17.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:17 smithi102 conmon[37544]: cluster 2024-01-13T09:54:15.861534+0000 mgr.smithi084.itrhok (mgr.14180) 290 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:17.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:17 smithi084 conmon[33408]: cluster 2024-01-13T09:54:15.861534+0000 mgr.smithi084.itrhok (mgr.14180) 290 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:19.323 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:54:19.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:19 smithi102 conmon[37544]: cluster 2024-01-13T09:54:17.861903+0000 mgr.smithi084.itrhok (mgr.14180) 291 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:19.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:19 smithi102 conmon[37544]: audit 2024-01-13T09 2024-01-13T09:54:19.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:19 smithi102 conmon[37544]: :54:19.327773+0000 mon.smithi084 (mon.0) 625 : audit [DBG] from='client.? 172.21.15.84:0/4113490802' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:54:19.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:19 smithi084 conmon[33408]: cluster 2024-01-13T09:54:17.861903+0000 mgr.smithi084.itrhok (mgr.14180) 291 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:19.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:19 smithi084 conmon[33408]: audit 2024-01-13T09:54:19.327773+0000 mon.smithi084 (mon.0) 625 : audit [DBG] from='client.? 172.21.15.84:0/4113490802' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:54:19.959 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:54:20.960 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:54:21.683 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:21 smithi084 conmon[33408]: cluster 2024-01-13T09:54:19.862289+0000 mgr.smithi084.itrhok (mgr.14180) 292 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:21.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:21 smithi102 conmon[37544]: cluster 2024-01-13T09:54:19.862289+0000 mgr.smithi084.itrhok (mgr.14180) 292 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:22.908 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:54:23.549 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:54:23.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:23 smithi102 conmon[37544]: cluster 2024-01-13T09:54:21.862766+0000 mgr.smithi084.itrhok (mgr.14180) 293 : cluster 2024-01-13T09:54:23.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:23 smithi102 conmon[37544]: [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:23.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:23 smithi102 conmon[37544]: audit 2024-01-13T09:54:22.912734+0000 mon.smithi084 (mon.0) 626 : audit [DBG] from='client.? 172.21.15.84:0/1572792353' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:54:23.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:23 smithi084 conmon[33408]: cluster 2024-01-13T09:54:21.862766+0000 mgr.smithi084.itrhok (mgr.14180) 293 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:23.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:23 smithi084 conmon[33408]: audit 2024-01-13T09:54:22.912734+0000 mon.smithi084 (mon.0) 626 : audit [DBG] from='client.? 172.21.15.84:0/1572792353' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:54:24.551 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:54:25.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:25 smithi102 conmon[37544]: cluster 2024-01-13T09:54:23.863164+0000 mgr.smithi084.itrhok (mgr.14180) 294 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:25.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:25 smithi084 conmon[33408]: cluster 2024-01-13T09:54:23. 2024-01-13T09:54:25.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:25 smithi084 conmon[33408]: 863164+0000 mgr.smithi084.itrhok (mgr.14180) 294 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:26.498 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:54:27.125 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:54:27.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:27 smithi102 conmon[37544]: cluster 2024-01-13T09:54:25.863591+0000 mgr.smithi084.itrhok (mgr.14180) 295 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:27.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:27 smithi102 conmon[37544]: audit 2024-01-13T09:54:26.502794+0000 mon.smithi084 (mon.0) 627 : audit [DBG] from='client.? 172.21.15.84:0/653640602' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:54:27.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:27 smithi084 conmon[33408]: cluster 2024-01-13T09:54:25.863591+0000 mgr.smithi084.itrhok (mgr.14180) 295 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:27.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:27 smithi084 conmon[33408]: audit 2024-01-13T09:54:26.502794+0000 mon.smithi084 (mon.0) 627 : audit [DBG] from='client.? 172.21.15.84:0/653640602' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:54:28.126 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:54:29.658 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:29 smithi084 conmon[33408]: cluster 2024-01-13T09:54:27.864058+0000 mgr.smithi084.itrhok 2024-01-13T09:54:29.658 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:29 smithi084 conmon[33408]: (mgr.14180) 296 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:29.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:29 smithi102 conmon[37544]: cluster 2024-01-13T09:54:27.864058+0000 mgr.smithi084.itrhok (mgr.14180) 296 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:30.037 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:54:30.639 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:54:30.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:30 smithi102 conmon[37544]: audit 2024-01-13T09:54:30.042184+0000 mon.smithi084 (mon.0) 628 : audit [DBG] from='client.? 172.21.15.84:0/2946152011' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:54:30.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:30 smithi084 conmon[33408]: audit 2024-01-13T09:54:30.042184+0000 mon.smithi084 (mon.0) 628 : audit [DBG] from='client.? 172.21.15.84:0/2946152011' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:54:31.640 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:54:31.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:31 smithi102 conmon[37544]: cluster 2024-01-13T09:54:29.864461+0000 mgr.smithi084.itrhok (mgr.14180) 297 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:31.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:31 smithi084 conmon[33408]: cluster 2024-01-13T09:54:29.864461+0000 2024-01-13T09:54:31.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:31 smithi084 conmon[33408]: mgr.smithi084.itrhok (mgr.14180) 297 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:33.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:33 smithi084 conmon[33408]: cluster 2024-01-13T09:54:31.864807+0000 mgr.smithi084.itrhok (mgr.14180) 298 : cluster 2024-01-13T09:54:33.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:33 smithi084 conmon[33408]: [DBG] pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:33.635 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:54:33.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:33 smithi102 conmon[37544]: cluster 2024-01-13T09:54:31.864807+0000 mgr.smithi084.itrhok (mgr.14180) 298 2024-01-13T09:54:33.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:33 smithi102 conmon[37544]: : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:34.250 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:54:34.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:34 smithi084 conmon[33408]: audit 2024-01-13T09:54:33.639906+0000 mon.smithi084 (mon.0) 629 : audit [DBG] from='client.? 172.21.15.84:0/326588984' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:54:34.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:34 smithi102 conmon[37544]: audit 2024-01-13T09:54:33.639906+0000 mon.smithi084 (mon.0) 629 : audit [DBG] from='client.? 172.21.15.84:0/326588984' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:54:35.252 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:54:35.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:35 smithi102 conmon[37544]: cluster 2024-01-13T09:54:33.865290+0000 mgr.smithi084.itrhok (mgr.14180) 299 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:35.747 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:35 smithi084 conmon[33408]: cluster 2024-01-13T09:54:33.865290+0000 mgr.smithi084.itrhok (mgr.14180) 299 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:35.747 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:35 smithi084 conmon[33408]: 2024-01-13T09:54:37.136 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:54:37.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:37 smithi102 conmon[37544]: cluster 2024-01-13T09:54:35.865812+0000 2024-01-13T09:54:37.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:37 smithi102 conmon[37544]: mgr.smithi084.itrhok (mgr.14180) 300 : cluster [DBG] pgmap v266: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:37.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:37 smithi102 conmon[37544]: audit 2024-01-13T09:54:37.140899+0000 mon.smithi084 (mon.0) 630 : audit 2024-01-13T09:54:37.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:37 smithi102 conmon[37544]: [DBG] from='client.? 172.21.15.84:0/497625507' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:54:37.755 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:37 smithi084 conmon[33408]: cluster 2024-01-13T09:54:35.865812 2024-01-13T09:54:37.755 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:37 smithi084 conmon[33408]: +0000 mgr.smithi084.itrhok (mgr.14180) 300 : cluster [DBG] pgmap v266: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:37.755 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:37 smithi084 conmon[33408]: audit 2024-01-13T09:54:37.140899+0000 mon.smithi084 (mon.0) 630 : audit [DBG] from='client.? 172.21.15.84:0/497625507' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:54:37.772 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:54:38.773 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:54:39.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:39 smithi084 conmon[33408]: cluster 2024-01-13T09:54:37.866258+0000 mgr.smithi084.itrhok (mgr.14180) 301 : cluster [DBG] pgmap v267: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:39.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:39 smithi102 conmon[37544]: cluster 2024-01-13T09:54:37.866258+0000 mgr.smithi084.itrhok (mgr.14180) 301 : cluster [DBG] pgmap v267: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:40.648 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:54:41.273 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:54:41.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:41 smithi084 conmon[33408]: cluster 2024-01-13T09:54:39.866675+0000 mgr.smithi084.itrhok (mgr.14180) 302 : cluster [DBG] pgmap v268: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:41.532 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:41 smithi084 conmon[33408]: audit 2024-01-13T09:54:40.652863+0000 mon.smithi084 (mon.0) 631 : audit [DBG] from='client.? 172.21.15.84:0/1594010147' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:54:41.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:41 smithi102 conmon[37544]: cluster 2024-01-13T09:54:39.866675+0000 mgr.smithi084.itrhok (mgr.14180) 302 : cluster [DBG] pgmap v268: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:41.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:41 smithi102 conmon[37544]: audit 2024-01-13T09:54:40.652863 2024-01-13T09:54:41.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:41 smithi102 conmon[37544]: +0000 mon.smithi084 (mon.0) 631 : audit [DBG] from='client.? 172.21.15.84:0/1594010147' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:54:42.274 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:54:42.712 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:42 smithi084 conmon[33408]: audit 2024-01-13T09:54:41.855715 2024-01-13T09:54:42.713 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:42 smithi084 conmon[33408]: +0000 mon.smithi084 (mon.0) 632 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T09:54:42.713 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:42 smithi084 conmon[33408]: audit 2024-01-13T09:54:42.170778+0000 mon.smithi084 (mon.0) 633 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:54:42.713 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:42 smithi084 conmon[33408]: audit 2024-01-13T09:54:42.171423+0000 mon.smithi084 (mon.0) 634 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T09:54:42.713 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:42 smithi084 conmon[33408]: audit 2024-01-13T09:54:42.175282+0000 mon.smithi084 (mon.0) 635 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:54:42.713 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:42 smithi084 conmon[33408]: audit 2024-01-13T09:54:42.178599+0000 mon.smithi084 (mon.0) 636 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T09:54:42.713 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:42 smithi084 conmon[33408]: audit 2024-01-13T09:54:42.181280+0000 mon.smithi084 (mon.0) 637 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:54:42.713 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:42 smithi084 conmon[33408]: audit 2024-01-13T09:54:42.182091+0000 mon.smithi084 (mon.0) 638 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:54:42.713 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:42 smithi084 conmon[33408]: audit 2024-01-13T09:54:42.182239+0000 mon.smithi084 (mon.0) 639 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:54:42.713 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:42 smithi084 conmon[33408]: audit 2024-01-13T09:54:42.182633+0000 mon.smithi084 (mon.0) 640 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:54:42.762 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:42 smithi102 conmon[37544]: audit 2024-01-13T09:54:41.855715+0000 mon.smithi084 (mon. 2024-01-13T09:54:42.762 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:42 smithi102 conmon[37544]: 0) 632 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T09:54:42.762 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:42 smithi102 conmon[37544]: audit 2024-01-13T09:54:42.170778+0000 mon.smithi084 (mon.0) 633 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:54:42.763 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:42 smithi102 conmon[37544]: audit 2024-01-13T09:54:42.171423+0000 mon.smithi084 (mon.0) 634 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T09:54:42.763 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:42 smithi102 conmon[37544]: audit 2024-01-13T09:54:42.175282+0000 mon.smithi084 (mon.0) 635 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' 2024-01-13T09:54:42.763 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:42 smithi102 conmon[37544]: audit 2024-01-13T09:54:42.178599+0000 mon.smithi084 (mon.0) 636 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T09:54:42.763 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:42 smithi102 conmon[37544]: audit 2024-01-13T09:54: 2024-01-13T09:54:42.763 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:42 smithi102 conmon[37544]: 42.181280+0000 mon.smithi084 (mon.0) 637 : audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:54:42.763 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:42 smithi102 conmon[37544]: audit 2024-01- 2024-01-13T09:54:42.763 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:42 smithi102 conmon[37544]: 13T09:54:42.182091+0000 mon.smithi084 (mon.0) 638 : 2024-01-13T09:54:42.763 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:42 smithi102 conmon[37544]: audit [INF] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:54:42.763 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:42 smithi102 conmon[37544]: audit 2024-01- 2024-01-13T09:54:42.763 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:42 smithi102 conmon[37544]: 13T09:54:42.182239+0000 2024-01-13T09:54:42.763 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:42 smithi102 conmon[37544]: mon.smithi084 (mon.0) 639 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:54:42.763 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:42 smithi102 conmon[37544]: 2024-01-13T09:54:42.763 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:42 smithi102 conmon[37544]: audit 2024-01-13T09:54:42.763 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:42 smithi102 conmon[37544]: 2024-01-13T09:54:42.182633 2024-01-13T09:54:42.764 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:42 smithi102 conmon[37544]: +0000 mon.smithi084 (mon.0) 640 : audit [DBG] 2024-01-13T09:54:42.764 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:42 smithi102 conmon[37544]: from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:54:43.702 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:43 smithi084 conmon[33408]: cluster 2024-01-13T09:54:41.867107+0000 mgr.smithi084.itrhok (mgr.14180) 303 : cluster [DBG] pgmap v269: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:43.702 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:43 smithi084 conmon[33408]: 2024-01-13T09:54:43.702 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:43 smithi084 conmon[33408]: cluster 2024-01-13T09:54:42.175603+0000 mgr.smithi084.itrhok (mgr.14180) 304 : cluster [DBG] pgmap v270: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:43.702 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:43 smithi084 conmon[33408]: cluster 2024-01-13T09:54:42.494881+0000 mon.smithi084 (mon.0) 641 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T09:54:43.702 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:43 smithi084 conmon[33408]: cluster 2024-01-13T09:54:42.494919+0000 mon.smithi084 (mon.0) 642 : cluster [INF] Cluster is now healthy 2024-01-13T09:54:43.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:43 smithi102 conmon[37544]: cluster 2024-01-13T09:54:41 2024-01-13T09:54:43.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:43 smithi102 conmon[37544]: .867107+0000 mgr.smithi084.itrhok (mgr.14180) 303 : cluster [DBG] pgmap v269: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:43.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:43 smithi102 conmon[37544]: cluster 2024-01-13T09:54:42.175603+0000 mgr.smithi084.itrhok (mgr.14180) 304 : cluster [DBG] pgmap v270: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:43.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:43 smithi102 conmon[37544]: cluster 2024-01-13T09:54:42.494881+0000 mon.smithi084 (mon.0) 641 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T09:54:43.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:43 smithi102 conmon[37544]: cluster 2024-01-13T09:54:42.494919+0000 mon.smithi084 (mon.0) 642 : cluster [INF] Cluster is now healthy 2024-01-13T09:54:45.200 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:54:45.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:45 smithi084 conmon[33408]: cluster 2024-01-13T09:54:44.175914+0000 mgr.smithi084.itrhok (mgr.14180) 305 : cluster [DBG] pgmap v271: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:45.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:45 smithi084 conmon[33408]: audit 2024-01-13T09:54:44.751950+0000 mon.smithi102 (mon.1) 16 : audit [INF] from='client.? 172.21.15.102:0/1921405296' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4af43a95-491c-4298-8b45-67de344446e7"}]: dispatch 2024-01-13T09:54:45.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:45 smithi084 conmon[33408]: audit 2024-01-13T09:54:44.752389+0000 mon.smithi084 (mon.0) 643 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4af43a95-491c-4298-8b45-67de344446e7"}]: dispatch 2024-01-13T09:54:45.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:45 smithi084 conmon[33408]: audit 2024-01-13T09:54:44.760325+0000 mon.smithi084 (mon.0) 644 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4af43a95-491c-4298-8b45-67de344446e7"}]': finished 2024-01-13T09:54:45.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:45 smithi084 conmon[33408]: cluster 2024-01-13T09:54:44.760369+0000 mon.smithi084 (mon.0) 645 : cluster [DBG] osdmap e34: 1 total, 0 up, 1 in 2024-01-13T09:54:45.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:45 smithi084 conmon[33408]: audit 2024-01-13T09:54:44.760484+0000 mon.smithi084 (mon.0) 646 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:54:45.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:45 smithi084 conmon[33408]: audit 2024-01-13T09:54:45.048494+0000 mon.smithi102 (mon.1) 17 : audit [INF] from='client.? 172.21.15.102:0/3200375460' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:54:45.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:45 smithi084 conmon[33408]: audit 2024-01-13T09:54:45.048880+0000 mon.smithi084 (mon.0) 647 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:54:45.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:45 smithi084 conmon[33408]: audit 2024-01-13T09:54:45.056793+0000 mon.smithi084 (mon.0) 648 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:54:45.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:45 smithi084 conmon[33408]: cluster 2024-01-13T09:54:45.056840+0000 mon.smithi084 (mon.0) 649 : cluster 2024-01-13T09:54:45.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:45 smithi084 conmon[33408]: [DBG] osdmap e35: 0 total, 0 up, 0 in 2024-01-13T09:54:45.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:45 smithi084 conmon[33408]: audit 2024-01-13T09:54:45.203173+0000 mon.smithi084 (mon.0) 650 : audit [DBG] from='client.? 172.21.15.84:0/3982781463' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:54:45.785 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:45 smithi102 conmon[37544]: cluster 2024-01-13T09:54:44.175914+0000 mgr.smithi084.itrhok (mgr.14180) 305 : cluster [DBG] pgmap v271: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:45.785 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:45 smithi102 conmon[37544]: audit 2024-01-13T09:54:44.751950+0000 mon.smithi102 (mon.1) 16 : audit [INF] from='client.? 172.21.15.102:0/1921405296' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4af43a95-491c-4298-8b45-67de344446e7"}]: dispatch 2024-01-13T09:54:45.785 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:45 smithi102 conmon[37544]: audit 2024-01-13T09:54:44.752389+0000 mon.smithi084 (mon.0) 643 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4af43a95-491c-4298-8b45-67de344446e7"}]: dispatch 2024-01-13T09:54:45.785 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:45 smithi102 conmon[37544]: audit 2024-01-13T09:54:44.760325+0000 mon.smithi084 (mon.0) 644 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4af43a95-491c-4298-8b45-67de344446e7"}]': finished 2024-01-13T09:54:45.785 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:45 smithi102 conmon[37544]: cluster 2024-01-13T09:54:44.760369+0000 mon.smithi084 (mon.0) 645 : cluster [DBG] osdmap e34: 1 total, 0 up, 1 in 2024-01-13T09:54:45.785 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:45 smithi102 conmon[37544]: audit 2024-01-13T09:54:44.760484+0000 mon.smithi084 (mon.0) 646 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:54:45.785 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:45 smithi102 conmon[37544]: audit 2024-01-13T09:54:45.048494+0000 mon.smithi102 (mon.1) 17 : audit [INF] from='client.? 172.21.15.102:0/3200375460' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:54:45.785 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:45 smithi102 conmon[37544]: audit 2024-01-13T09:54:45.048880+0000 mon.smithi084 (mon.0) 647 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:54:45.785 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:45 smithi102 conmon[37544]: audit 2024-01-13T09:54:45.056793+0000 mon.smithi084 (mon.0) 648 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:54:45.785 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:45 smithi102 conmon[37544]: cluster 2024-01-13T09:54:45.056840+0000 mon.smithi084 (mon.0) 649 : cluster [DBG] osdmap e35: 0 total, 0 up, 0 in 2024-01-13T09:54:45.785 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:45 smithi102 conmon[37544]: audit 2024-01-13T09:54:45.203173+0000 mon.smithi084 (mon.0) 650 : audit [DBG] from='client.? 172.21.15.84:0/3982781463' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:54:45.834 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":35,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:54:46.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:46 smithi084 conmon[33408]: audit 2024-01-13T09:54:45.576250+0000 mon.smithi084 (mon.0) 651 : audit [INF] from='client.? 172.21.15.84:0/3914778292' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "277b1827-dbb9-4992-a2a1-0508cde9b26a"}]: dispatch 2024-01-13T09:54:46.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:46 smithi084 conmon[33408]: audit 2024-01-13T09:54:45.579623+0000 mon.smithi084 (mon.0) 652 : audit [INF] from='client.? 172.21.15.84:0/3914778292' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "277b1827-dbb9-4992-a2a1-0508cde9b26a"}]': finished 2024-01-13T09:54:46.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:46 smithi084 conmon[33408]: cluster 2024-01-13T09:54:45.579660 2024-01-13T09:54:46.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:46 smithi084 conmon[33408]: +0000 mon.smithi084 (mon.0) 653 : cluster [DBG] osdmap e36: 1 total, 0 up, 1 in 2024-01-13T09:54:46.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:46 smithi084 conmon[33408]: audit 2024- 2024-01-13T09:54:46.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:46 smithi084 conmon[33408]: 01-13T09:54:45.579787+0000 mon.smithi084 (mon.0 2024-01-13T09:54:46.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:46 smithi084 conmon[33408]: ) 654 : audit [DBG] from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:54:46.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:46 smithi084 conmon[33408]: audit 2024-01-13T09:54:46.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:46 smithi084 conmon[33408]: 2024-01-13 2024-01-13T09:54:46.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:46 smithi084 conmon[33408]: T09:54:45.876125+0000 2024-01-13T09:54:46.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:46 smithi084 conmon[33408]: mon.smithi084 (mon.0) 655 : audit [INF] 2024-01-13T09:54:46.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:46 smithi084 conmon[33408]: from='client.? 172.21.15.84:0/1672506046' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:54:46.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:46 smithi084 conmon[33408]: audit 2024-01-13T09:54:46.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:46 smithi084 conmon[33408]: 2024-01-13T09:54: 2024-01-13T09:54:46.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:46 smithi084 conmon[33408]: 45.880145+0000 mon.smithi084 (mon. 2024-01-13T09:54:46.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:46 smithi084 conmon[33408]: 0) 656 : audit [INF] from='client.? 172.21.15.84:0/1672506046' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:54:46.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:46 smithi084 conmon[33408]: 2024-01-13T09:54:46.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:46 smithi084 conmon[33408]: cluster 2024-01- 2024-01-13T09:54:46.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:46 smithi084 conmon[33408]: 13T09:54:45.880198+0000 2024-01-13T09:54:46.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:46 smithi084 conmon[33408]: mon.smithi084 (mon 2024-01-13T09:54:46.784 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:46 smithi084 conmon[33408]: .0) 657 : cluster [DBG] 2024-01-13T09:54:46.785 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:46 smithi084 conmon[33408]: osdmap e37: 0 total, 0 up, 0 in 2024-01-13T09:54:46.835 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:54:46.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:46 smithi102 conmon[37544]: audit 2024-01-13T09:54:45.576250+0000 mon.smithi084 (mon.0) 651 : audit 2024-01-13T09:54:46.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:46 smithi102 conmon[37544]: [INF] from='client.? 172.21.15.84:0/3914778292' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "277b1827-dbb9-4992-a2a1-0508cde9b26a"}]: dispatch 2024-01-13T09:54:46.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:46 smithi102 conmon[37544]: audit 2024-01-13T09:54:45.579623+0000 mon.smithi084 (mon.0) 652 : audit [INF] from='client.? 172.21.15.84:0/3914778292' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "277b1827-dbb9-4992-a2a1-0508cde9b26a"}]': finished 2024-01-13T09:54:46.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:46 smithi102 conmon[37544]: cluster 2024-01-13T09:54:45.579660+0000 mon.smithi084 (mon.0) 2024-01-13T09:54:46.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:46 smithi102 conmon[37544]: 653 : cluster [DBG] osdmap e36: 1 total, 0 up, 1 in 2024-01-13T09:54:46.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:46 smithi102 conmon[37544]: audit 2024-01-13T09:54: 2024-01-13T09:54:46.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:46 smithi102 conmon[37544]: 45.579787+0000 mon.smithi084 (mon.0) 654 : audit [DBG] 2024-01-13T09:54:46.990 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:46 smithi102 conmon[37544]: from='mgr.14180 172.21.15.84:0/1382161807' entity='mgr.smithi084.itrhok' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:54:46.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:46 smithi102 conmon[37544]: audit 2024-01-13T09:54:45.876125+0000 mon.smithi084 2024-01-13T09:54:46.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:46 smithi102 conmon[37544]: (mon.0) 655 : audit [INF] from='client.? 172.21.15.84:0/1672506046' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:54:46.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:46 smithi102 conmon[37544]: audit 2024-01- 2024-01-13T09:54:46.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:46 smithi102 conmon[37544]: 13T09:54:45.880145+0000 mon.smithi084 (mon.0 2024-01-13T09:54:46.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:46 smithi102 conmon[37544]: ) 656 : audit [INF] from='client.? 172.21.15.84:0/1672506046' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:54:46.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:46 smithi102 conmon[37544]: cluster 2024-01-13T09: 2024-01-13T09:54:46.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:46 smithi102 conmon[37544]: 54:45.880198+0000 mon.smithi084 (mon.0) 657 : 2024-01-13T09:54:46.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:46 smithi102 conmon[37544]: cluster [DBG] osdmap e37: 0 total, 0 up, 0 in 2024-01-13T09:54:47.713 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: cephadm 2024-01-13T09:54:45.811326+0000 mgr.smithi084.itrhok (mgr.14180) 306 : cephadm [ERR] executing create_from_spec_one(([('smithi084', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi102', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:54:47.713 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:54:47.713 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:54:47.713 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: return f(*arg) 2024-01-13T09:54:47.713 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:54:47.713 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:54:47.713 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:54:47.713 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: code, '\n'.join(err))) 2024-01-13T09:54:47.714 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:54:47.714 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp3lbdb6zq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpaccd48ka:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:54:47.714 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:54:47.714 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:54:47.714 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:54:47.714 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4af43a95-491c-4298-8b45-67de344446e7 2024-01-13T09:54:47.714 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:54:47.714 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:54:47.714 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:54:47.714 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:54:47.715 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:54:47.715 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr self.prepare() 2024-01-13T09:54:47.715 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:54:47.715 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:54:47.715 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:54:47.715 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:54:47.715 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:54:47.715 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:54:47.715 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:54:47.715 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:54:47.715 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:54:47.715 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:54:47.715 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:54:47.715 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:54:47.715 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:54:47.716 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:54:47.716 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:54:47.716 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:54:47.716 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:54:47.716 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:54:47.716 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:54:47.716 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:54:47.716 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:54:47.716 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:54:47.716 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:54:47.716 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:54:47.716 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:54:47.716 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:54:47.716 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:54:47.717 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:54:47.717 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:54:47.717 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:54:47.717 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:54:47.717 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:54:47.717 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:54:47.717 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:54:47.717 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:54:47.717 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:54:47.717 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:54:47.717 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:54:47.717 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:54:47.717 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:54:47.717 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:54:47.717 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:54:47.718 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:54:47.718 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:54:47.718 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:54:47.718 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:54:47.718 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:54:47.718 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:54:47.718 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:54:47.718 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:54:47.718 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:54:47.718 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:54:47.718 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: main() 2024-01-13T09:54:47.718 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:54:47.718 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: r = ctx.func(ctx) 2024-01-13T09:54:47.718 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:54:47.719 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:54:47.719 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:54:47.719 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:54:47.719 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:54:47.719 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:54:47.719 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-8777 2024-01-13T09:54:47.719 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: 4f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:54:47.719 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:54:47.719 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:54:47.719 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:54:47.719 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:54:47.719 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:54:47.719 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp3lbdb6zq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpaccd48ka:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:54:47.719 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: cluster 2024-01-13T09:54:46.176211+0000 mgr.smithi084.itrhok (mgr.14180) 307 : cluster [DBG] pgmap v276: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:47.719 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: cephadm 2024-01-13T09:54:46.606874+0000 mgr.smithi084.itrhok (mgr.14180) 308 : cephadm [ERR] executing create_from_spec_one(([('smithi084', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi102', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:54:47.720 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:54:47.720 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:54:47.720 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: return f(*arg) 2024-01-13T09:54:47.720 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:54:47.720 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:54:47.720 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:54:47.720 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: code, '\n'.join(err))) 2024-01-13T09:54:47.720 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi084/config 2024-01-13T09:54:47.720 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmphyr5pt61:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp0s1z_6if:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:54:47.720 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:54:47.720 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:54:47.720 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:54:47.720 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 277b1827-dbb9-4992-a2a1-0508cde9b26a 2024-01-13T09:54:47.720 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:54:47.721 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:54:47.721 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:54:47.721 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:54:47.721 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:54:47.721 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr self.prepare() 2024-01-13T09:54:47.721 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:54:47.721 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:54:47.721 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:54:47.721 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:54:47.721 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:54:47.721 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:54:47.721 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:54:47.721 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:54:47.721 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:54:47.721 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:54:47.722 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:54:47.722 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:54:47.722 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:54:47.722 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:54:47.722 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:54:47.722 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:54:47.722 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:54:47.722 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:54:47.722 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:54:47.723 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:54:47.723 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:54:47.723 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:54:47.724 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:54:47.724 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:54:47.724 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:54:47.724 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:54:47.725 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:54:47.725 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:54:47.725 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:54:47.725 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:54:47.725 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:54:47.725 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:54:47.725 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:54:47.725 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:54:47.725 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:54:47.725 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:54:47.725 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:54:47.725 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:54:47.725 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr rol 2024-01-13T09:54:47.726 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: lback_osd(self.args, self.osd_id) 2024-01-13T09:54:47.726 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:54:47.726 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:54:47.726 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:54:47.726 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:54:47.726 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:54:47.726 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:54:47.726 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:54:47.726 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:54:47.726 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:54:47.726 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:54:47.726 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:54:47.726 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:54:47.726 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:54:47.726 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: main() 2024-01-13T09:54:47.726 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:54:47.727 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: r = ctx.func(ctx) 2024-01-13T09:54:47.727 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:54:47.727 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:54:47.727 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:54:47.727 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:54:47.727 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:54:47.727 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:54:47.727 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:54:47.727 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:54:47.727 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:54:47.727 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:54:47.727 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:54:47.727 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:54:47.727 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmphyr5pt61:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp0s1z_6if:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:54:47.727 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: cephadm 2024-01-13T09:54:46.608076+0000 mgr.smithi084.itrhok (mgr.14180) 309 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T09:54:47.727 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: service_id: all-available-devices 2024-01-13T09:54:47.728 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: service_name: osd.all-available-devices 2024-01-13T09:54:47.728 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: placement: 2024-01-13T09:54:47.728 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: host_pattern: '*' 2024-01-13T09:54:47.728 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: spec: 2024-01-13T09:54:47.728 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: data_devices: 2024-01-13T09:54:47.728 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: all: true 2024-01-13T09:54:47.728 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: filter_logic: AND 2024-01-13T09:54:47.728 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: objectstore: bluestore 2024-01-13T09:54:47.728 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:54:47.728 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp3lbdb6zq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpaccd48ka:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:54:47.728 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:54:47.728 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:54:47.728 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:54:47.728 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4af43a95-491c-4298-8b45-67de344446e7 2024-01-13T09:54:47.729 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:54:47.729 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:54:47.729 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:54:47.729 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:54:47.729 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:54:47.729 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr self.prepare() 2024-01-13T09:54:47.729 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:54:47.729 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:54:47.729 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:54:47.729 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:54:47.729 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:54:47.729 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:54:47.729 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:54:47.729 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:54:47.729 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:54:47.729 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:54:47.730 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:54:47.730 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:54:47.730 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:54:47.730 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:54:47.730 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:54:47.730 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:54:47.730 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:54:47.730 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:54:47.730 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:54:47.730 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:54:47.730 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:54:47.730 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", 2024-01-13T09:54:47.730 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: line 194, in dispatch 2024-01-13T09:54:47.730 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:54:47.730 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:54:47.731 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:54:47.731 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:54:47.731 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:54:47.731 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:54:47.731 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:54:47.731 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:54:47.731 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:54:47.731 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:54:47.731 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:54:47.731 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:54:47.731 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:54:47.731 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:54:47.731 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:54:47.731 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:54:47.731 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:54:47.732 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:54:47.732 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:54:47.732 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:54:47.732 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:54:47.732 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:54:47.732 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:54:47.732 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:54:47.732 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:54:47.732 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:54:47.732 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:54:47.732 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:54:47.732 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:54:47.732 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:54:47.732 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: main() 2024-01-13T09:54:47.733 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:54:47.733 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: r = ctx.func(ctx) 2024-01-13T09:54:47.733 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:54:47.733 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:54:47.733 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:54:47.733 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:54:47.733 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:54:47.733 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:54:47.733 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:54:47.733 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:54:47.733 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:54:47.733 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:54:47.733 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:54:47.733 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:54:47.734 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp3lbdb6zq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpaccd48ka:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:54:47.734 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:54:47.734 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T09:54:47.734 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: if self._apply_service(spec): 2024-01-13T09:54:47.734 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T09:54:47.734 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T09:54:47.734 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T09:54:47.734 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T09:54:47.734 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T09:54:47.734 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T09:54:47.734 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T09:54:47.734 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T09:54:47.734 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T09:54:47.734 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: raise self._value 2024-01-13T09:54:47.735 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T09:54:47.735 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: result = (True, func(*args, **kwds)) 2024-01-13T09:54:47.735 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T09:54:47.735 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: return list(map(*args)) 2024-01-13T09:54:47.735 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:54:47.735 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: return f(*arg) 2024-01-13T09:54:47.735 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:54:47.735 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:54:47.735 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:54:47.735 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: code, '\n'.join(err))) 2024-01-13T09:54:47.735 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:54:47.735 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp3lbdb6zq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpaccd48ka:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:54:47.735 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:54:47.735 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:54:47.736 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:54:47.736 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name clie 2024-01-13T09:54:47.736 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: nt.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4af43a95-491c-4298-8b45-67de344446e7 2024-01-13T09:54:47.736 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:54:47.736 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:54:47.736 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:54:47.736 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:54:47.736 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:54:47.736 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr self.prepare() 2024-01-13T09:54:47.736 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:54:47.736 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:54:47.736 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:54:47.736 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:54:47.736 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:54:47.736 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:54:47.737 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:54:47.737 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:54:47.737 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:54:47.737 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:54:47.737 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:54:47.737 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr 2024-01-13T09:54:47.737 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:54:47.737 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:54:47.737 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:54:47.737 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:54:47.737 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:54:47.737 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:54:47.737 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:54:47.737 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:54:47.737 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:54:47.738 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:54:47.738 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:54:47.738 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:54:47.738 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:54:47.738 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:54:47.738 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr instance.main() 2024-01-13T09:54:47.738 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:54:47.738 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:54:47.738 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:54:47.738 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:54:47.738 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:54:47.738 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:54:47.738 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:54:47.738 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:54:47.738 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:54:47.739 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:54:47.739 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:54:47.739 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:54:47.739 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:54:47.739 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:54:47.739 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:54:47.739 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:54:47.739 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:54:47.739 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:54:47.739 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:54:47.739 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:54:47.739 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:54:47.739 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:54:47.739 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:54:47.739 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: Traceback (most recent call last): 2024-01-13T09:54:47.739 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:54:47.740 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: main() 2024-01-13T09:54:47.740 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:54:47.740 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: r = ctx.func(ctx) 2024-01-13T09:54:47.740 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:54:47.740 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:54:47.740 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:54:47.740 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:54:47.740 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:54:47.740 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:54:47.740 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:54:47.740 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: return func(ctx) 2024-01-13T09:54:47.740 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:54:47.740 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:54:47.740 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:54:47.741 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:54:47.741 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp3lbdb6zq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpaccd48ka:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:54:47.741 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: cluster 2024-01-13T09:54:46.608689+0000 mgr.smithi084.itrhok (mgr.14180) 310 : cluster [DBG] pgmap v277: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:47.741 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:47 smithi084 conmon[33408]: cluster 2024-01-13T09:54:46.878218+0000 mon.smithi084 (mon.0) 658 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T09:54:47.991 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: cephadm 2024-01-13T09:54:45.811326+0000 mgr.smithi084.itrhok (mgr.14180) 306 : cephadm [ERR] executing create_from_spec_one(([('smithi084', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi102', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:54:47.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:54:47.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:54:47.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: return f(*arg) 2024-01-13T09:54:47.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:54:47.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:54:47.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:54:47.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: code, '\n'.join(err))) 2024-01-13T09:54:47.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:54:47.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp3lbdb6zq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpaccd48ka:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:54:47.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:54:47.992 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:54:47.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:54:47.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4af43a95-491c-4298-8b45-67de344446e7 2024-01-13T09:54:47.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:54:47.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:54:47.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:54:47.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:54:47.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:54:47.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr self.prepare() 2024-01-13T09:54:47.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:54:47.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:54:47.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:54:47.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:54:47.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:54:47.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:54:47.993 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:54:47.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:54:47.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:54:47.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:54:47.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:54:47.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:54:47.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:54:47.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:54:47.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:54:47.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:54:47.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:54:47.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:54:47.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:54:47.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:54:47.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:54:47.994 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:54:47.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:54:47.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:54:47.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:54:47.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:54:47.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:54:47.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:54:47.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:54:47.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:54:47.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:54:47.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:54:47.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:54:47.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:54:47.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:54:47.995 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:54:47.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:54:47.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:54:47.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:54:47.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:54:47.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:54:47.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:54:47.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:54:47.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:54:47.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:54:47.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:54:47.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:54:47.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:54:47.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:54:47.996 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:54:47.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:54:47.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:54:47.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: main() 2024-01-13T09:54:47.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:54:47.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: r = ctx.func(ctx) 2024-01-13T09:54:47.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:54:47.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:54:47.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:54:47.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:54:47.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:54:47.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:54:47.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-8777 2024-01-13T09:54:47.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: 4f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:54:47.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:54:47.997 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:54:47.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:54:47.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:54:47.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:54:47.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp3lbdb6zq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpaccd48ka:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:54:47.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: cluster 2024-01-13T09:54:46.176211+0000 mgr.smithi084.itrhok (mgr.14180) 307 : cluster [DBG] pgmap v276: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:47.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: cephadm 2024-01-13T09:54:46.606874+0000 mgr.smithi084.itrhok (mgr.14180) 308 : cephadm [ERR] executing create_from_spec_one(([('smithi084', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi102', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:54:47.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:54:47.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:54:47.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: return f(*arg) 2024-01-13T09:54:47.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:54:47.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:54:47.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:54:47.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: code, '\n'.join(err))) 2024-01-13T09:54:47.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi084/config 2024-01-13T09:54:47.998 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmphyr5pt61:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp0s1z_6if:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:54:47.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:54:47.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:54:47.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:54:47.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 277b1827-dbb9-4992-a2a1-0508cde9b26a 2024-01-13T09:54:47.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:54:47.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:54:47.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:54:47.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:54:47.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:54:47.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr self.prepare() 2024-01-13T09:54:47.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:54:47.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:54:47.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:54:47.999 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:54:48.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:54:48.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:54:48.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:54:48.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:54:48.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:54:48.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:54:48.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:54:48.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:54:48.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:54:48.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:54:48.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:54:48.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:54:48.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:54:48.000 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:54:48.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:54:48.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:54:48.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:54:48.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:54:48.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:54:48.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:54:48.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:54:48.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:54:48.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:54:48.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:54:48.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:54:48.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:54:48.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:54:48.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:54:48.001 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:54:48.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:54:48.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:54:48.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:54:48.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:54:48.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:54:48.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr rol 2024-01-13T09:54:48.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: lback_osd(self.args, self.osd_id) 2024-01-13T09:54:48.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:54:48.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:54:48.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:54:48.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:54:48.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:54:48.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:54:48.002 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:54:48.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:54:48.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:54:48.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:54:48.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:54:48.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:54:48.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:54:48.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: main() 2024-01-13T09:54:48.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:54:48.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: r = ctx.func(ctx) 2024-01-13T09:54:48.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:54:48.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:54:48.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:54:48.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:54:48.003 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:54:48.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:54:48.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:54:48.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:54:48.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:54:48.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:54:48.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:54:48.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:54:48.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmphyr5pt61:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp0s1z_6if:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:54:48.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: cephadm 2024-01-13T09:54:46.608076+0000 mgr.smithi084.itrhok (mgr.14180) 309 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T09:54:48.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: service_id: all-available-devices 2024-01-13T09:54:48.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: service_name: osd.all-available-devices 2024-01-13T09:54:48.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: placement: 2024-01-13T09:54:48.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: host_pattern: '*' 2024-01-13T09:54:48.004 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: spec: 2024-01-13T09:54:48.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: data_devices: 2024-01-13T09:54:48.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: all: true 2024-01-13T09:54:48.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: filter_logic: AND 2024-01-13T09:54:48.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: objectstore: bluestore 2024-01-13T09:54:48.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:54:48.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp3lbdb6zq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpaccd48ka:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:54:48.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:54:48.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:54:48.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:54:48.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4af43a95-491c-4298-8b45-67de344446e7 2024-01-13T09:54:48.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:54:48.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:54:48.005 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:54:48.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:54:48.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:54:48.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr self.prepare() 2024-01-13T09:54:48.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:54:48.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:54:48.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:54:48.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:54:48.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:54:48.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:54:48.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:54:48.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:54:48.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:54:48.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:54:48.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:54:48.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:54:48.006 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:54:48.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:54:48.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:54:48.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:54:48.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:54:48.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:54:48.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:54:48.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:54:48.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:54:48.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", 2024-01-13T09:54:48.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: line 194, in dispatch 2024-01-13T09:54:48.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:54:48.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:54:48.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:54:48.007 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:54:48.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:54:48.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:54:48.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:54:48.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:54:48.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:54:48.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:54:48.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:54:48.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:54:48.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:54:48.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:54:48.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:54:48.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:54:48.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:54:48.008 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:54:48.009 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:54:48.009 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:54:48.009 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:54:48.009 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:54:48.009 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:54:48.009 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:54:48.009 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:54:48.009 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:54:48.009 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:54:48.009 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:54:48.009 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:54:48.009 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:54:48.009 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: main() 2024-01-13T09:54:48.009 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:54:48.009 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: r = ctx.func(ctx) 2024-01-13T09:54:48.010 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:54:48.010 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:54:48.010 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:54:48.010 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:54:48.010 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:54:48.010 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:54:48.010 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:54:48.010 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:54:48.010 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:54:48.010 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:54:48.010 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:54:48.010 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:54:48.010 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp3lbdb6zq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpaccd48ka:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:54:48.010 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:54:48.011 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T09:54:48.011 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: if self._apply_service(spec): 2024-01-13T09:54:48.011 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T09:54:48.011 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T09:54:48.011 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T09:54:48.011 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T09:54:48.011 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T09:54:48.011 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T09:54:48.011 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T09:54:48.011 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T09:54:48.011 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T09:54:48.011 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: raise self._value 2024-01-13T09:54:48.011 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T09:54:48.011 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: result = (True, func(*args, **kwds)) 2024-01-13T09:54:48.011 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T09:54:48.012 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: return list(map(*args)) 2024-01-13T09:54:48.012 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:54:48.012 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: return f(*arg) 2024-01-13T09:54:48.012 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:54:48.012 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:54:48.012 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:54:48.012 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: code, '\n'.join(err))) 2024-01-13T09:54:48.012 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/mon.smithi102/config 2024-01-13T09:54:48.012 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp3lbdb6zq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpaccd48ka:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:54:48.012 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:54:48.012 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:54:48.012 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:54:48.012 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name clie 2024-01-13T09:54:48.013 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: nt.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4af43a95-491c-4298-8b45-67de344446e7 2024-01-13T09:54:48.013 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:54:48.013 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:54:48.013 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:54:48.013 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:54:48.013 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:54:48.014 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr self.prepare() 2024-01-13T09:54:48.014 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:54:48.014 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:54:48.014 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:54:48.014 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:54:48.014 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:54:48.014 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:54:48.014 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:54:48.014 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:54:48.014 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:54:48.014 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:54:48.014 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:54:48.014 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr 2024-01-13T09:54:48.014 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:54:48.015 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:54:48.015 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:54:48.015 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:54:48.015 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:54:48.015 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:54:48.015 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:54:48.015 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:54:48.015 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:54:48.015 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:54:48.015 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:54:48.015 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:54:48.015 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:54:48.015 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:54:48.015 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr instance.main() 2024-01-13T09:54:48.015 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:54:48.016 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:54:48.016 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:54:48.016 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:54:48.016 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:54:48.016 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:54:48.016 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:54:48.016 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:54:48.016 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:54:48.016 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:54:48.016 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:54:48.016 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:54:48.016 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:54:48.016 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:54:48.017 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:54:48.017 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:54:48.017 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:54:48.017 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:54:48.017 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:54:48.017 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:54:48.017 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:54:48.017 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:54:48.017 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:54:48.017 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: Traceback (most recent call last): 2024-01-13T09:54:48.017 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:54:48.017 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: main() 2024-01-13T09:54:48.017 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:54:48.018 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: r = ctx.func(ctx) 2024-01-13T09:54:48.018 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:54:48.018 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:54:48.018 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:54:48.018 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:54:48.018 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:54:48.018 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:54:48.018 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:54:48.018 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: return func(ctx) 2024-01-13T09:54:48.018 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:54:48.018 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:54:48.018 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: File "/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:54:48.018 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:54:48.018 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi102 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp3lbdb6zq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpaccd48ka:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:54:48.019 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: cluster 2024-01-13T09:54:46.608689+0000 mgr.smithi084.itrhok (mgr.14180) 310 : cluster [DBG] pgmap v277: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:48.019 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:47 smithi102 conmon[37544]: cluster 2024-01-13T09:54:46.878218+0000 mon.smithi084 (mon.0) 658 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T09:54:48.781 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:54:49.453 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":37,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:54:49.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:49 smithi084 conmon[33408]: cluster 2024-01-13T09:54:48.609089+0000 mgr.smithi084.itrhok (mgr.14180) 311 : cluster [DBG] pgmap v278: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:49.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:49 smithi084 conmon[33408]: audit 2024-01-13T09:54:48.786072+0000 mon.smithi084 (mon.0) 659 : audit [DBG] from='client.? 172.21.15.84:0/1694026201' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:54:49.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:49 smithi102 conmon[37544]: cluster 2024-01-13T09:54:48.609089+0000 mgr.smithi084.itrhok (mgr.14180) 311 : cluster [DBG] pgmap v278: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:49.989 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:49 smithi102 conmon[37544]: audit 2024-01-13T09:54:48.786072+0000 mon.smithi084 (mon.0) 659 : audit [DBG] from='client.? 172.21.15.84:0/1694026201' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:54:50.454 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:54:51.736 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:51 smithi084 conmon[33408]: cluster 2024-01-13T09:54:50.609481+0000 mgr.smithi084.itrhok (mgr.14180) 312 : cluster [DBG] pgmap v279: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:51.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:51 smithi102 conmon[37544]: cluster 2024-01-13T09:54:50.609481+0000 mgr.smithi084.itrhok (mgr.14180) 312 : cluster [DBG] pgmap v279: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:52.358 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-13T09:54:52.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:52 smithi102 conmon[37544]: audit 2024-01-13T09:54:52.361761+0000 mon.smithi084 (mon.0) 660 : audit [DBG] from='client.? 172.21.15.84:0/1952395383' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:54:52.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:52 smithi084 conmon[33408]: audit 2024-01-13T09:54:52.361761+0000 mon.smithi084 (mon.0) 660 : audit [DBG] from='client.? 172.21.15.84:0/1952395383' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:54:52.782 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:52 smithi084 conmon[33408]: 2024-01-13T09:54:53.012 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":37,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705139279,"num_remapped_pgs":0} 2024-01-13T09:54:53.012 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 779, in ceph_osds while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (121) after waiting for 120 seconds 2024-01-13T09:54:53.013 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-13T09:54:53.044 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-13T09:54:53.098 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-01-13T09:54:53.098 DEBUG:teuthology.orchestra.run.smithi084:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-13T09:54:53.115 DEBUG:teuthology.orchestra.run.smithi102:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-13T09:54:53.134 INFO:tasks.cephadm:Stopping all daemons... 2024-01-13T09:54:53.134 INFO:tasks.cephadm.mon.smithi084:Stopping mon.smithi084... 2024-01-13T09:54:53.134 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-503cf380-b1f8-11ee-95ac-87774f69a715@mon.smithi084 2024-01-13T09:54:53.282 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:53 smithi084 systemd[1]: Stopping Ceph mon.smithi084 for 503cf380-b1f8-11ee-95ac-87774f69a715... 2024-01-13T09:54:53.282 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:53 smithi084 bash[101324]: Error: no container with name or ID ceph-503cf380-b1f8-11ee-95ac-87774f69a715-mon.smithi084 found: no such container 2024-01-13T09:54:53.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:53 smithi102 conmon[37544]: cluster 2024-01-13T09:54:52.609990+0000 mgr.smithi084.itrhok (mgr.14180) 313 : cluster [DBG] pgmap v280: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:53.753 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:53 smithi084 conmon[33408]: cluster 2024-01-13T09:54:52.609990+0000 mgr.smithi084.itrhok (mgr.14180) 313 : cluster [DBG] pgmap v280: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:54:53.754 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:53 smithi084 conmon[33408]: debug 2024-01-13T09:54:53.499+0000 7f0be1021700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.smithi084 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-13T09:54:53.754 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 13 09:54:53 smithi084 conmon[33408]: debug 2024-01-13T09:54:53.499+0000 7f0be1021700 -1 mon.smithi084@0(leader) e2 *** Got Signal Terminated *** 2024-01-13T09:54:54.370 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-503cf380-b1f8-11ee-95ac-87774f69a715@mon.smithi084.service' 2024-01-13T09:54:54.425 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-13T09:54:54.425 INFO:tasks.cephadm.mon.smithi084:Stopped mon.smithi084 2024-01-13T09:54:54.426 INFO:tasks.cephadm.mon.smithi102:Stopping mon.smithi102... 2024-01-13T09:54:54.426 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-503cf380-b1f8-11ee-95ac-87774f69a715@mon.smithi102 2024-01-13T09:54:54.739 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:54 smithi102 systemd[1]: Stopping Ceph mon.smithi102 for 503cf380-b1f8-11ee-95ac-87774f69a715... 2024-01-13T09:54:54.740 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:54 smithi102 bash[45691]: Error: no container with name or ID ceph-503cf380-b1f8-11ee-95ac-87774f69a715-mon.smithi102 found: no such container 2024-01-13T09:54:55.076 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:54 smithi102 conmon[37544]: debug 2024-01-13T09:54:54.745+0000 7fc07c767700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.smithi102 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-13T09:54:55.076 INFO:journalctl@ceph.mon.smithi102.smithi102.stdout:Jan 13 09:54:54 smithi102 conmon[37544]: debug 2024-01-13T09:54:54.745+0000 7fc07c767700 -1 mon.smithi102@1(peon) e2 *** Got Signal Terminated *** 2024-01-13T09:54:55.644 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-503cf380-b1f8-11ee-95ac-87774f69a715@mon.smithi102.service' 2024-01-13T09:54:55.690 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-13T09:54:55.691 INFO:tasks.cephadm.mon.smithi102:Stopped mon.smithi102 2024-01-13T09:54:55.691 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 --force --keep-logs 2024-01-13T09:55:02.591 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 --force --keep-logs 2024-01-13T09:55:06.301 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-13T09:55:06.340 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-13T09:55:06.371 INFO:tasks.cephadm:Archiving crash dumps... 2024-01-13T09:55:06.373 DEBUG:teuthology.misc:Transferring archived files from smithi084:/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515860/remote/ubuntu@smithi084.front.sepia.ceph.com/crash 2024-01-13T09:55:06.374 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash -- . 2024-01-13T09:55:06.423 INFO:teuthology.orchestra.run.smithi084.stderr:tar: /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash: Cannot open: No such file or directory 2024-01-13T09:55:06.423 INFO:teuthology.orchestra.run.smithi084.stderr:tar: Error is not recoverable: exiting now 2024-01-13T09:55:06.425 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515860/remote/ubuntu@smithi102.front.sepia.ceph.com/crash 2024-01-13T09:55:06.426 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash -- . 2024-01-13T09:55:06.463 INFO:teuthology.orchestra.run.smithi102.stderr:tar: /var/lib/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/crash: Cannot open: No such file or directory 2024-01-13T09:55:06.463 INFO:teuthology.orchestra.run.smithi102.stderr:tar: Error is not recoverable: exiting now 2024-01-13T09:55:06.464 INFO:tasks.cephadm:Checking cluster log for badness... 2024-01-13T09:55:06.464 DEBUG:teuthology.orchestra.run.smithi084:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-01-13T09:55:06.494 INFO:teuthology.orchestra.run.smithi084.stderr:grep: /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/ceph.log: No such file or directory 2024-01-13T09:55:06.496 INFO:tasks.cephadm:Compressing logs... 2024-01-13T09:55:06.496 DEBUG:teuthology.orchestra.run.smithi084:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-13T09:55:06.539 DEBUG:teuthology.orchestra.run.smithi102:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-13T09:55:06.565 INFO:teuthology.orchestra.run.smithi084.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-01-13T09:55:06.565 INFO:teuthology.orchestra.run.smithi084.stderr:: No such file or directory 2024-01-13T09:55:06.565 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/ceph-mgr.smithi084.itrhok.log 2024-01-13T09:55:06.565 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/ceph-mon.smithi084.log 2024-01-13T09:55:06.565 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/ceph-volume.log 2024-01-13T09:55:06.567 INFO:teuthology.orchestra.run.smithi102.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-01-13T09:55:06.567 INFO:teuthology.orchestra.run.smithi102.stderr:: No such file or directory 2024-01-13T09:55:06.567 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/ceph-volume.log 2024-01-13T09:55:06.567 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/ceph-mgr.smithi102.yjovaz.log 2024-01-13T09:55:06.568 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/ceph-mon.smithi102.log 2024-01-13T09:55:06.569 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/ceph-volume.log: /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/ceph-mgr.smithi102.yjovaz.log: /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/ceph-mon.smithi102.log: 92.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-13T09:55:06.570 INFO:teuthology.orchestra.run.smithi102.stderr: 91.8% -- replaced with /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/ceph-mgr.smithi102.yjovaz.log.gz 2024-01-13T09:55:06.571 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/ceph-mgr.smithi084.itrhok.log: /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/ceph-mon.smithi084.log: /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/ceph-volume.log: 94.0% -- replaced with /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/ceph-volume.log.gz 2024-01-13T09:55:06.572 INFO:teuthology.orchestra.run.smithi102.stderr: 94.0% -- replaced with /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/ceph-volume.log.gz 2024-01-13T09:55:06.573 INFO:teuthology.orchestra.run.smithi084.stderr: 92.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-13T09:55:06.643 INFO:teuthology.orchestra.run.smithi102.stderr: 92.6% -- replaced with /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/ceph-mon.smithi102.log.gz 2024-01-13T09:55:06.645 INFO:teuthology.orchestra.run.smithi102.stderr: 2024-01-13T09:55:06.645 INFO:teuthology.orchestra.run.smithi102.stderr:real 0m0.090s 2024-01-13T09:55:06.645 INFO:teuthology.orchestra.run.smithi102.stderr:user 0m0.091s 2024-01-13T09:55:06.645 INFO:teuthology.orchestra.run.smithi102.stderr:sys 0m0.018s 2024-01-13T09:55:06.650 INFO:teuthology.orchestra.run.smithi084.stderr: 91.0% -- replaced with /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/ceph-mgr.smithi084.itrhok.log.gz 2024-01-13T09:55:06.810 INFO:teuthology.orchestra.run.smithi084.stderr: 89.9% -- replaced with /var/log/ceph/503cf380-b1f8-11ee-95ac-87774f69a715/ceph-mon.smithi084.log.gz 2024-01-13T09:55:06.811 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-01-13T09:55:06.811 INFO:teuthology.orchestra.run.smithi084.stderr:real 0m0.259s 2024-01-13T09:55:06.812 INFO:teuthology.orchestra.run.smithi084.stderr:user 0m0.336s 2024-01-13T09:55:06.812 INFO:teuthology.orchestra.run.smithi084.stderr:sys 0m0.031s 2024-01-13T09:55:06.812 INFO:tasks.cephadm:Archiving logs... 2024-01-13T09:55:06.812 DEBUG:teuthology.misc:Transferring archived files from smithi084:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515860/remote/ubuntu@smithi084.front.sepia.ceph.com/log 2024-01-13T09:55:06.813 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-13T09:55:06.959 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515860/remote/ubuntu@smithi102.front.sepia.ceph.com/log 2024-01-13T09:55:06.960 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-13T09:55:07.004 INFO:tasks.cephadm:Removing cluster... 2024-01-13T09:55:07.004 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 --force 2024-01-13T09:55:07.436 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 503cf380-b1f8-11ee-95ac-87774f69a715 --force 2024-01-13T09:55:07.889 INFO:tasks.cephadm:Removing cephadm ... 2024-01-13T09:55:07.889 DEBUG:teuthology.orchestra.run.smithi084:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-13T09:55:07.907 DEBUG:teuthology.orchestra.run.smithi102:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-13T09:55:07.924 INFO:tasks.cephadm:Teardown complete 2024-01-13T09:55:07.924 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 779, in ceph_osds while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (121) after waiting for 120 seconds 2024-01-13T09:55:08.137 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=79bd745d0b5647a6841d43cc73227cc6 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 779, in ceph_osds while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (121) after waiting for 120 seconds 2024-01-13T09:55:08.140 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-01-13T09:55:08.151 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2024-01-13T09:55:08.182 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi084:/dev/vg_nvme/lv_1... 2024-01-13T09:55:08.182 DEBUG:teuthology.orchestra.run.smithi084:> sudo nvme disconnect -n lv_1 2024-01-13T09:55:08.317 INFO:teuthology.orchestra.run.smithi084.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-01-13T09:55:08.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T09:55:08.319 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi084:/dev/vg_nvme/lv_2... 2024-01-13T09:55:08.319 DEBUG:teuthology.orchestra.run.smithi084:> sudo nvme disconnect -n lv_2 2024-01-13T09:55:08.436 INFO:teuthology.orchestra.run.smithi084.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-01-13T09:55:08.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T09:55:08.437 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi084:/dev/vg_nvme/lv_3... 2024-01-13T09:55:08.437 DEBUG:teuthology.orchestra.run.smithi084:> sudo nvme disconnect -n lv_3 2024-01-13T09:55:08.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T09:55:08.560 INFO:teuthology.orchestra.run.smithi084.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-01-13T09:55:08.560 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi084:/dev/vg_nvme/lv_4... 2024-01-13T09:55:08.560 DEBUG:teuthology.orchestra.run.smithi084:> sudo nvme disconnect -n lv_4 2024-01-13T09:55:08.698 INFO:teuthology.orchestra.run.smithi084.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-01-13T09:55:08.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T09:55:08.699 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-01-13T09:55:08.699 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/scratch_devs 2024-01-13T09:55:08.726 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi102:/dev/vg_nvme/lv_1... 2024-01-13T09:55:08.726 DEBUG:teuthology.orchestra.run.smithi102:> sudo nvme disconnect -n lv_1 2024-01-13T09:55:08.848 INFO:teuthology.orchestra.run.smithi102.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-01-13T09:55:08.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T09:55:08.850 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi102:/dev/vg_nvme/lv_2... 2024-01-13T09:55:08.850 DEBUG:teuthology.orchestra.run.smithi102:> sudo nvme disconnect -n lv_2 2024-01-13T09:55:08.974 INFO:teuthology.orchestra.run.smithi102.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-01-13T09:55:08.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T09:55:08.976 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi102:/dev/vg_nvme/lv_3... 2024-01-13T09:55:08.976 DEBUG:teuthology.orchestra.run.smithi102:> sudo nvme disconnect -n lv_3 2024-01-13T09:55:09.096 INFO:teuthology.orchestra.run.smithi102.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-01-13T09:55:09.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T09:55:09.098 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi102:/dev/vg_nvme/lv_4... 2024-01-13T09:55:09.098 DEBUG:teuthology.orchestra.run.smithi102:> sudo nvme disconnect -n lv_4 2024-01-13T09:55:09.215 INFO:teuthology.orchestra.run.smithi102.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-01-13T09:55:09.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T09:55:09.217 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-01-13T09:55:09.217 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/scratch_devs 2024-01-13T09:55:09.250 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-13T09:55:09.259 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-13T09:55:09.259 DEBUG:teuthology.orchestra.run.smithi084:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-13T09:55:09.265 DEBUG:teuthology.orchestra.run.smithi102:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-13T09:55:09.279 INFO:teuthology.orchestra.run.smithi084.stderr:bash: ntpq: command not found 2024-01-13T09:55:09.284 INFO:teuthology.orchestra.run.smithi084.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-13T09:55:09.284 INFO:teuthology.orchestra.run.smithi084.stdout:=============================================================================== 2024-01-13T09:55:09.284 INFO:teuthology.orchestra.run.smithi084.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 29 -669us[ -688us] +/- 68ms 2024-01-13T09:55:09.284 INFO:teuthology.orchestra.run.smithi084.stdout:^* hv02.front.sepia.ceph.com 3 6 377 28 -967us[ -986us] +/- 57ms 2024-01-13T09:55:09.285 INFO:teuthology.orchestra.run.smithi084.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 28 +1904us[+1885us] +/- 75ms 2024-01-13T09:55:09.285 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T09:55:09.309 INFO:teuthology.orchestra.run.smithi102.stderr:bash: ntpq: command not found 2024-01-13T09:55:09.315 INFO:teuthology.orchestra.run.smithi102.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-13T09:55:09.315 INFO:teuthology.orchestra.run.smithi102.stdout:=============================================================================== 2024-01-13T09:55:09.315 INFO:teuthology.orchestra.run.smithi102.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 31 -608us[ -648us] +/- 68ms 2024-01-13T09:55:09.315 INFO:teuthology.orchestra.run.smithi102.stdout:^* hv02.front.sepia.ceph.com 3 6 377 31 -909us[ -949us] +/- 57ms 2024-01-13T09:55:09.315 INFO:teuthology.orchestra.run.smithi102.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 30 +1950us[+1950us] +/- 75ms 2024-01-13T09:55:09.315 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T09:55:09.316 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-13T09:55:09.325 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-13T09:55:09.325 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-13T09:55:09.358 DEBUG:teuthology.orchestra.run.smithi084:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-13T09:55:09.423 DEBUG:teuthology.orchestra.run.smithi102:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-13T09:55:09.492 DEBUG:teuthology.orchestra.run.smithi084:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-13T09:55:09.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T09:55:09.523 DEBUG:teuthology.orchestra.run.smithi102:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-13T09:55:09.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T09:55:09.555 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-13T09:55:09.564 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-13T09:55:09.609 INFO:teuthology.task.internal:Duration was 1234.102134 seconds 2024-01-13T09:55:09.609 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-13T09:55:09.710 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-13T09:55:09.710 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-13T09:55:09.713 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-13T09:55:09.758 INFO:teuthology.orchestra.run.smithi084.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-13T09:55:09.762 INFO:teuthology.orchestra.run.smithi102.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-13T09:55:10.045 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-13T09:55:10.056 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi084.front.sepia.ceph.com 2024-01-13T09:55:10.057 DEBUG:teuthology.orchestra.run.smithi084:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-13T09:55:10.079 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi102.front.sepia.ceph.com 2024-01-13T09:55:10.079 DEBUG:teuthology.orchestra.run.smithi102:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-13T09:55:10.102 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-13T09:55:10.102 DEBUG:teuthology.orchestra.run.smithi084:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-13T09:55:10.121 DEBUG:teuthology.orchestra.run.smithi102:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-13T09:55:10.197 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-13T09:55:10.198 DEBUG:teuthology.orchestra.run.smithi084:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-13T09:55:10.200 DEBUG:teuthology.orchestra.run.smithi102:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-13T09:55:10.345 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-13T09:55:10.356 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-13T09:55:10.356 DEBUG:teuthology.orchestra.run.smithi084:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-13T09:55:10.383 DEBUG:teuthology.orchestra.run.smithi102:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-13T09:55:10.411 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-13T09:55:10.421 DEBUG:teuthology.orchestra.run.smithi084:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-13T09:55:10.425 DEBUG:teuthology.orchestra.run.smithi102:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-13T09:55:10.448 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern = core 2024-01-13T09:55:10.477 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern = core 2024-01-13T09:55:10.496 DEBUG:teuthology.orchestra.run.smithi084:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-13T09:55:10.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T09:55:10.535 DEBUG:teuthology.orchestra.run.smithi102:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-13T09:55:10.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T09:55:10.552 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-13T09:55:10.562 INFO:teuthology.task.internal:Transferring archived files... 2024-01-13T09:55:10.563 DEBUG:teuthology.misc:Transferring archived files from smithi084:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515860/remote/smithi084 2024-01-13T09:55:10.564 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-13T09:55:10.621 DEBUG:teuthology.misc:Transferring archived files from smithi102:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515860/remote/smithi102 2024-01-13T09:55:10.622 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-13T09:55:10.662 INFO:teuthology.task.internal:Removing archive directory... 2024-01-13T09:55:10.662 DEBUG:teuthology.orchestra.run.smithi084:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-13T09:55:10.664 DEBUG:teuthology.orchestra.run.smithi102:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-13T09:55:10.712 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-13T09:55:10.724 INFO:teuthology.task.internal:Not uploading archives. 2024-01-13T09:55:10.725 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-13T09:55:10.743 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-13T09:55:10.743 DEBUG:teuthology.orchestra.run.smithi084:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-13T09:55:10.746 DEBUG:teuthology.orchestra.run.smithi102:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-13T09:55:10.760 INFO:teuthology.orchestra.run.smithi084.stdout: 262179 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 13 09:55 /home/ubuntu/cephtest 2024-01-13T09:55:10.768 INFO:teuthology.orchestra.run.smithi102.stdout: 262179 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 13 09:55 /home/ubuntu/cephtest 2024-01-13T09:55:10.770 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-13T09:55:10.780 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-13T09:55:10.842 INFO:teuthology.nuke:Checking targets against current locks 2024-01-13T09:55:10.868 DEBUG:teuthology.nuke:shortname: smithi084 2024-01-13T09:55:10.869 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-13T09:55:10.894 DEBUG:teuthology.nuke:shortname: smithi102 2024-01-13T09:55:10.894 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-13T09:55:10.897 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi084.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515860', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-13 09:29:25.372783', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB3VZXVJ3zgggvP/1bs8JLYarhJK4t2BfPZfKm5zL56Nc6E0uXQRjD0vHFw+UIgyz6eYMiH8a50HG6MBLpL5Yr4='} 2024-01-13T09:55:10.922 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi102.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515860', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-13 09:29:25.371384', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJK2s6PnhQ1UyCQIe+xYA2Q3/b8STcSd/k2D6xm/EUEbCCrF+MAbmPmHWTxTnuqDC11N05b67hwZNZjGxpTskn0='} 2024-01-13T09:55:10.951 INFO:teuthology.orchestra.console.smithi084:Power off 2024-01-13T09:55:10.951 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-13T09:55:10.974 DEBUG:teuthology.orchestra.console.smithi084:power off output: Chassis Power Control: Down/Off 2024-01-13T09:55:10.974 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-13T09:55:10.994 INFO:teuthology.orchestra.console.smithi102:Power off 2024-01-13T09:55:10.994 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-13T09:55:11.003 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is on 2024-01-13T09:55:11.018 DEBUG:teuthology.orchestra.console.smithi102:power off output: Chassis Power Control: Down/Off 2024-01-13T09:55:11.019 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-13T09:55:11.047 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is on 2024-01-13T09:55:15.005 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-13T09:55:15.048 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-13T09:55:15.116 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is on 2024-01-13T09:55:15.159 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is on 2024-01-13T09:55:19.121 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-13T09:55:19.160 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-13T09:55:19.231 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is off 2024-01-13T09:55:19.271 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is off 2024-01-13T09:55:19.333 INFO:teuthology.orchestra.console.smithi084:Power off completed 2024-01-13T09:55:19.372 INFO:teuthology.orchestra.console.smithi102:Power off completed 2024-01-13T09:55:19.477 INFO:teuthology.run:Summary data: description: rados/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} duration: 1234.102133512497 failure_reason: reached maximum tries (121) after waiting for 120 seconds owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=79bd745d0b5647a6841d43cc73227cc6 status: fail success: false 2024-01-13T09:55:19.477 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-13T09:55:19.606 INFO:teuthology.run:FAIL