2024-01-13T11:35:44.706 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-13T11:35:44.706 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --description rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} --verbose --name yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7516049 -- /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7516049/orig.config.yaml 2024-01-13T11:35:44.739 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-13T11:35:44.821 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7516049 branch: wip-yuri2-testing-2024-01-12-1128-pacific description: rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} email: yweinste@redhat.com first_in_suite: false job_id: '7516049' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri2-testing-2024-01-12-1128-pacific ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 445562ab4bc3ddfb386936119050695810860bcb ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 445562ab4bc3ddfb386936119050695810860bcb kernel: hwe: true workunit: branch: wip-yuri2-testing-2024-01-12-1128-pacific sha1: 445562ab4bc3ddfb386936119050695810860bcb owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - - host.b - client.1 seed: 5246 sha1: 445562ab4bc3ddfb386936119050695810860bcb sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri2-testing-2024-01-12-1128-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 445562ab4bc3ddfb386936119050695810860bcb targets: smithi088.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= smithi138.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - nvme_loop: null - cephadm: roleless: true - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - vip: null - cephadm.shell: host.a: - ceph orch device ls --refresh - vip.exec: all-hosts: - systemctl stop nfs-server - cephadm.shell: host.a: - ceph fs volume create foofs - cephadm.apply: specs: - placement: count: 2 service_id: foo service_type: nfs spec: port: 12049 - service_id: nfs.foo service_type: ingress spec: backend_service: nfs.foo frontend_port: 2049 monitor_port: 9002 virtual_ip: '{{VIP0}}/{{VIPPREFIXLEN}}' - cephadm.wait_for_service: service: nfs.foo - cephadm.wait_for_service: service: ingress.nfs.foo - cephadm.shell: host.a: - ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake - vip.exec: host.a: - mkdir /mnt/foo - sleep 5 - mount -t nfs {{VIP0}}:/fake /mnt/foo - echo test > /mnt/foo/testfile - sync - cephadm.shell: host.a: - "echo \"Check with each haproxy down in turn...\"\nfor haproxy in `ceph orch\ \ ps | grep ^haproxy.nfs.foo. | awk '{print $1}'`; do\n ceph orch daemon stop\ \ $haproxy\n while ! ceph orch ps | grep $haproxy | grep stopped; do sleep\ \ 1 ; done\n cat /mnt/foo/testfile\n echo $haproxy > /mnt/foo/testfile\n \ \ sync\n ceph orch daemon start $haproxy\n while ! ceph orch ps | grep $haproxy\ \ | grep running; do sleep 1 ; done\ndone\n" volumes: - /mnt/foo:/mnt/foo - cephadm.shell: host.a: - stat -c '%u %g' /var/log/ceph | grep '167 167' - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch ls | grep '^osd.all-available-devices ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-12_21:52:25 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-01-13T11:35:44.822 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa; will attempt to use it 2024-01-13T11:35:44.822 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks 2024-01-13T11:35:44.823 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-13T11:35:44.824 INFO:teuthology.task.internal:Checking packages... 2024-01-13T11:35:44.847 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '445562ab4bc3ddfb386936119050695810860bcb' 2024-01-13T11:35:44.848 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-13T11:35:44.848 INFO:teuthology.packaging:ref: None 2024-01-13T11:35:44.848 INFO:teuthology.packaging:tag: None 2024-01-13T11:35:44.848 INFO:teuthology.packaging:branch: wip-yuri2-testing-2024-01-12-1128-pacific 2024-01-13T11:35:44.848 INFO:teuthology.packaging:sha1: 445562ab4bc3ddfb386936119050695810860bcb 2024-01-13T11:35:44.848 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri2-testing-2024-01-12-1128-pacific 2024-01-13T11:35:44.994 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-456-g445562ab-1focal 2024-01-13T11:35:44.995 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-13T11:35:45.001 INFO:teuthology.task.internal:no buildpackages task found 2024-01-13T11:35:45.001 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-13T11:35:45.008 INFO:teuthology.task.internal:Saving configuration 2024-01-13T11:35:45.021 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-13T11:35:45.027 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-13T11:35:45.050 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7516049', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-13 11:29:03.519793', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-01-13T11:35:45.074 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi138.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7516049', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-13 11:29:03.518248', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-01-13T11:35:45.074 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-13T11:35:45.080 INFO:teuthology.task.internal:roles: ubuntu@smithi088.front.sepia.ceph.com - ['host.a', 'client.0'] 2024-01-13T11:35:45.080 INFO:teuthology.task.internal:roles: ubuntu@smithi138.front.sepia.ceph.com - ['host.b', 'client.1'] 2024-01-13T11:35:45.080 INFO:teuthology.run_tasks:Running task console_log... 2024-01-13T11:35:45.146 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa23c5c93a0>, signals=[15]) 2024-01-13T11:35:45.146 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-13T11:35:45.152 INFO:teuthology.task.internal:Opening connections... 2024-01-13T11:35:45.152 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi088.front.sepia.ceph.com 2024-01-13T11:35:45.155 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-13T11:35:45.244 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi138.front.sepia.ceph.com 2024-01-13T11:35:45.247 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-13T11:35:45.333 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-13T11:35:45.340 DEBUG:teuthology.orchestra.run.smithi088:> uname -m 2024-01-13T11:35:45.347 INFO:teuthology.orchestra.run.smithi088.stdout:x86_64 2024-01-13T11:35:45.347 DEBUG:teuthology.orchestra.run.smithi088:> cat /etc/os-release 2024-01-13T11:35:45.395 INFO:teuthology.orchestra.run.smithi088.stdout:NAME="Ubuntu" 2024-01-13T11:35:45.395 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-01-13T11:35:45.395 INFO:teuthology.orchestra.run.smithi088.stdout:ID=ubuntu 2024-01-13T11:35:45.395 INFO:teuthology.orchestra.run.smithi088.stdout:ID_LIKE=debian 2024-01-13T11:35:45.395 INFO:teuthology.orchestra.run.smithi088.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-01-13T11:35:45.395 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION_ID="20.04" 2024-01-13T11:35:45.395 INFO:teuthology.orchestra.run.smithi088.stdout:HOME_URL="https://www.ubuntu.com/" 2024-01-13T11:35:45.395 INFO:teuthology.orchestra.run.smithi088.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-01-13T11:35:45.395 INFO:teuthology.orchestra.run.smithi088.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-01-13T11:35:45.395 INFO:teuthology.orchestra.run.smithi088.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-01-13T11:35:45.395 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION_CODENAME=focal 2024-01-13T11:35:45.395 INFO:teuthology.orchestra.run.smithi088.stdout:UBUNTU_CODENAME=focal 2024-01-13T11:35:45.396 INFO:teuthology.lock.ops:Updating smithi088.front.sepia.ceph.com on lock server 2024-01-13T11:35:45.416 DEBUG:teuthology.orchestra.run.smithi138:> uname -m 2024-01-13T11:35:45.423 INFO:teuthology.orchestra.run.smithi138.stdout:x86_64 2024-01-13T11:35:45.423 DEBUG:teuthology.orchestra.run.smithi138:> cat /etc/os-release 2024-01-13T11:35:45.473 INFO:teuthology.orchestra.run.smithi138.stdout:NAME="Ubuntu" 2024-01-13T11:35:45.473 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-01-13T11:35:45.474 INFO:teuthology.orchestra.run.smithi138.stdout:ID=ubuntu 2024-01-13T11:35:45.474 INFO:teuthology.orchestra.run.smithi138.stdout:ID_LIKE=debian 2024-01-13T11:35:45.474 INFO:teuthology.orchestra.run.smithi138.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-01-13T11:35:45.474 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION_ID="20.04" 2024-01-13T11:35:45.474 INFO:teuthology.orchestra.run.smithi138.stdout:HOME_URL="https://www.ubuntu.com/" 2024-01-13T11:35:45.474 INFO:teuthology.orchestra.run.smithi138.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-01-13T11:35:45.474 INFO:teuthology.orchestra.run.smithi138.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-01-13T11:35:45.474 INFO:teuthology.orchestra.run.smithi138.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-01-13T11:35:45.474 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION_CODENAME=focal 2024-01-13T11:35:45.474 INFO:teuthology.orchestra.run.smithi138.stdout:UBUNTU_CODENAME=focal 2024-01-13T11:35:45.474 INFO:teuthology.lock.ops:Updating smithi138.front.sepia.ceph.com on lock server 2024-01-13T11:35:45.499 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-13T11:35:45.507 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-13T11:35:45.512 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-13T11:35:45.512 DEBUG:teuthology.orchestra.run.smithi088:> test '!' -e /home/ubuntu/cephtest 2024-01-13T11:35:45.515 DEBUG:teuthology.orchestra.run.smithi138:> test '!' -e /home/ubuntu/cephtest 2024-01-13T11:35:45.520 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-13T11:35:45.526 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-13T11:35:45.527 DEBUG:teuthology.orchestra.run.smithi088:> test -z $(ls -A /var/lib/ceph) 2024-01-13T11:35:45.529 DEBUG:teuthology.orchestra.run.smithi138:> test -z $(ls -A /var/lib/ceph) 2024-01-13T11:35:45.548 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-13T11:35:45.603 INFO:teuthology.run_tasks:Running task kernel... 2024-01-13T11:35:45.618 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-01-13T11:35:45.618 INFO:teuthology.task.kernel:normalize config orig: {'hwe': True} 2024-01-13T11:35:45.618 DEBUG:teuthology.task.kernel:normalized overrides {'host.a': {'hwe': True}, 'client.0': {'hwe': True}, 'host.b': {'hwe': True}, 'client.1': {'hwe': True}} 2024-01-13T11:35:45.618 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.b': {'kdb': True, 'sha1': 'distro', 'hwe': True}}, timeout 300 2024-01-13T11:35:45.619 DEBUG:teuthology.orchestra.run.smithi088:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-13T11:35:45.619 DEBUG:teuthology.orchestra.run.smithi138:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-13T11:35:45.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T11:35:45.625 DEBUG:teuthology.orchestra.run.smithi088:> uname -r 2024-01-13T11:35:45.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T11:35:45.626 DEBUG:teuthology.orchestra.run.smithi138:> uname -r 2024-01-13T11:35:45.632 INFO:teuthology.orchestra.run.smithi088.stdout:5.4.0-124-generic 2024-01-13T11:35:45.632 INFO:teuthology.task.kernel:Running kernel on smithi088: 5.4.0-124-generic 2024-01-13T11:35:45.632 DEBUG:teuthology.orchestra.run.smithi088:> sudo apt-get clean 2024-01-13T11:35:45.634 INFO:teuthology.orchestra.run.smithi138.stdout:5.4.0-124-generic 2024-01-13T11:35:45.634 INFO:teuthology.task.kernel:Running kernel on smithi138: 5.4.0-124-generic 2024-01-13T11:35:45.634 DEBUG:teuthology.orchestra.run.smithi138:> sudo apt-get clean 2024-01-13T11:35:45.751 DEBUG:teuthology.orchestra.run.smithi088:> sudo apt-get update 2024-01-13T11:35:45.755 DEBUG:teuthology.orchestra.run.smithi138:> sudo apt-get update 2024-01-13T11:35:45.917 INFO:teuthology.orchestra.run.smithi088.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2024-01-13T11:35:46.109 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2024-01-13T11:35:46.110 INFO:teuthology.orchestra.run.smithi138.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2024-01-13T11:35:46.110 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2024-01-13T11:35:46.110 INFO:teuthology.orchestra.run.smithi138.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2024-01-13T11:35:46.110 INFO:teuthology.orchestra.run.smithi138.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2024-01-13T11:35:46.172 INFO:teuthology.orchestra.run.smithi088.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2024-01-13T11:35:46.208 INFO:teuthology.orchestra.run.smithi088.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [694 kB] 2024-01-13T11:35:46.311 INFO:teuthology.orchestra.run.smithi088.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,647 kB] 2024-01-13T11:35:46.369 INFO:teuthology.orchestra.run.smithi088.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [406 kB] 2024-01-13T11:35:46.375 INFO:teuthology.orchestra.run.smithi088.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [13.2 kB] 2024-01-13T11:35:46.375 INFO:teuthology.orchestra.run.smithi088.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [2,461 kB] 2024-01-13T11:35:46.407 INFO:teuthology.orchestra.run.smithi088.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [35.1 kB] 2024-01-13T11:35:46.407 INFO:teuthology.orchestra.run.smithi088.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [343 kB] 2024-01-13T11:35:46.411 INFO:teuthology.orchestra.run.smithi088.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [552 B] 2024-01-13T11:35:46.412 INFO:teuthology.orchestra.run.smithi088.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [637 kB] 2024-01-13T11:35:46.420 INFO:teuthology.orchestra.run.smithi088.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [921 kB] 2024-01-13T11:35:46.428 INFO:teuthology.orchestra.run.smithi138.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,647 kB] 2024-01-13T11:35:46.432 INFO:teuthology.orchestra.run.smithi088.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [193 kB] 2024-01-13T11:35:46.435 INFO:teuthology.orchestra.run.smithi088.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-01-13T11:35:46.441 INFO:teuthology.orchestra.run.smithi088.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,188 B] 2024-01-13T11:35:46.441 INFO:teuthology.orchestra.run.smithi088.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [23.9 kB] 2024-01-13T11:35:46.442 INFO:teuthology.orchestra.run.smithi088.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,796 B] 2024-01-13T11:35:46.442 INFO:teuthology.orchestra.run.smithi088.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [548 B] 2024-01-13T11:35:46.514 INFO:teuthology.orchestra.run.smithi088.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2024-01-13T11:35:46.569 INFO:teuthology.orchestra.run.smithi138.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [920 kB] 2024-01-13T11:35:46.590 INFO:teuthology.orchestra.run.smithi138.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [694 kB] 2024-01-13T11:35:46.600 INFO:teuthology.orchestra.run.smithi138.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [406 kB] 2024-01-13T11:35:46.605 INFO:teuthology.orchestra.run.smithi138.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [13.2 kB] 2024-01-13T11:35:46.606 INFO:teuthology.orchestra.run.smithi138.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [35.1 kB] 2024-01-13T11:35:46.607 INFO:teuthology.orchestra.run.smithi138.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [2,461 kB] 2024-01-13T11:35:46.646 INFO:teuthology.orchestra.run.smithi138.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [343 kB] 2024-01-13T11:35:46.646 INFO:teuthology.orchestra.run.smithi138.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [552 B] 2024-01-13T11:35:46.646 INFO:teuthology.orchestra.run.smithi138.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [921 kB] 2024-01-13T11:35:46.653 INFO:teuthology.orchestra.run.smithi088.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,030 kB] 2024-01-13T11:35:46.658 INFO:teuthology.orchestra.run.smithi138.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [637 kB] 2024-01-13T11:35:46.665 INFO:teuthology.orchestra.run.smithi138.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [193 kB] 2024-01-13T11:35:46.671 INFO:teuthology.orchestra.run.smithi138.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-01-13T11:35:46.674 INFO:teuthology.orchestra.run.smithi138.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,188 B] 2024-01-13T11:35:46.674 INFO:teuthology.orchestra.run.smithi138.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [23.9 kB] 2024-01-13T11:35:46.674 INFO:teuthology.orchestra.run.smithi138.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,796 B] 2024-01-13T11:35:46.675 INFO:teuthology.orchestra.run.smithi138.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,030 kB] 2024-01-13T11:35:46.697 INFO:teuthology.orchestra.run.smithi138.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [548 B] 2024-01-13T11:35:46.747 INFO:teuthology.orchestra.run.smithi138.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [489 kB] 2024-01-13T11:35:46.754 INFO:teuthology.orchestra.run.smithi138.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.2 kB] 2024-01-13T11:35:46.754 INFO:teuthology.orchestra.run.smithi138.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2,579 kB] 2024-01-13T11:35:46.786 INFO:teuthology.orchestra.run.smithi138.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [36.4 kB] 2024-01-13T11:35:46.786 INFO:teuthology.orchestra.run.smithi138.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [360 kB] 2024-01-13T11:35:46.792 INFO:teuthology.orchestra.run.smithi138.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [552 B] 2024-01-13T11:35:46.793 INFO:teuthology.orchestra.run.smithi138.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [763 kB] 2024-01-13T11:35:46.803 INFO:teuthology.orchestra.run.smithi138.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,144 kB] 2024-01-13T11:35:46.817 INFO:teuthology.orchestra.run.smithi138.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [274 kB] 2024-01-13T11:35:46.820 INFO:teuthology.orchestra.run.smithi138.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [25.7 kB] 2024-01-13T11:35:46.824 INFO:teuthology.orchestra.run.smithi138.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [26.1 kB] 2024-01-13T11:35:46.824 INFO:teuthology.orchestra.run.smithi138.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-01-13T11:35:46.824 INFO:teuthology.orchestra.run.smithi138.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,768 B] 2024-01-13T11:35:46.825 INFO:teuthology.orchestra.run.smithi138.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [620 B] 2024-01-13T11:35:46.841 INFO:teuthology.orchestra.run.smithi138.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-01-13T11:35:46.842 INFO:teuthology.orchestra.run.smithi138.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-01-13T11:35:46.842 INFO:teuthology.orchestra.run.smithi138.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-01-13T11:35:46.843 INFO:teuthology.orchestra.run.smithi138.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-01-13T11:35:46.843 INFO:teuthology.orchestra.run.smithi138.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-01-13T11:35:46.845 INFO:teuthology.orchestra.run.smithi138.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-01-13T11:35:47.038 INFO:teuthology.orchestra.run.smithi088.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [920 kB] 2024-01-13T11:35:47.067 INFO:teuthology.orchestra.run.smithi088.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [489 kB] 2024-01-13T11:35:47.080 INFO:teuthology.orchestra.run.smithi088.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.2 kB] 2024-01-13T11:35:47.080 INFO:teuthology.orchestra.run.smithi088.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2,579 kB] 2024-01-13T11:35:47.148 INFO:teuthology.orchestra.run.smithi088.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [36.4 kB] 2024-01-13T11:35:47.148 INFO:teuthology.orchestra.run.smithi088.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [360 kB] 2024-01-13T11:35:47.158 INFO:teuthology.orchestra.run.smithi088.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [552 B] 2024-01-13T11:35:47.321 INFO:teuthology.orchestra.run.smithi088.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,144 kB] 2024-01-13T11:35:47.963 INFO:teuthology.orchestra.run.smithi088.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [763 kB] 2024-01-13T11:35:48.004 INFO:teuthology.orchestra.run.smithi088.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [274 kB] 2024-01-13T11:35:48.014 INFO:teuthology.orchestra.run.smithi088.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [25.7 kB] 2024-01-13T11:35:48.015 INFO:teuthology.orchestra.run.smithi088.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-01-13T11:35:48.016 INFO:teuthology.orchestra.run.smithi088.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [26.1 kB] 2024-01-13T11:35:48.018 INFO:teuthology.orchestra.run.smithi088.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,768 B] 2024-01-13T11:35:48.018 INFO:teuthology.orchestra.run.smithi088.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [620 B] 2024-01-13T11:35:48.018 INFO:teuthology.orchestra.run.smithi088.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-01-13T11:35:48.019 INFO:teuthology.orchestra.run.smithi088.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-01-13T11:35:48.046 INFO:teuthology.orchestra.run.smithi088.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-01-13T11:35:48.210 INFO:teuthology.orchestra.run.smithi088.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-01-13T11:35:48.401 INFO:teuthology.orchestra.run.smithi088.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-01-13T11:35:48.426 INFO:teuthology.orchestra.run.smithi088.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-01-13T11:35:50.423 INFO:teuthology.orchestra.run.smithi088.stdout:Fetched 18.5 MB in 3s (7,126 kB/s) 2024-01-13T11:35:50.558 INFO:teuthology.orchestra.run.smithi138.stdout:Fetched 18.5 MB in 3s (6,616 kB/s) 2024-01-13T11:35:51.937 INFO:teuthology.orchestra.run.smithi088.stdout:Reading package lists... 2024-01-13T11:35:51.963 DEBUG:teuthology.orchestra.run.smithi088:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2024-01-13T11:35:52.042 INFO:teuthology.orchestra.run.smithi088.stdout:Reading package lists... 2024-01-13T11:35:52.094 INFO:teuthology.orchestra.run.smithi138.stdout:Reading package lists... 2024-01-13T11:35:52.122 DEBUG:teuthology.orchestra.run.smithi138:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2024-01-13T11:35:52.247 INFO:teuthology.orchestra.run.smithi088.stdout:Building dependency tree... 2024-01-13T11:35:52.247 INFO:teuthology.orchestra.run.smithi088.stdout:Reading state information... 2024-01-13T11:35:52.253 INFO:teuthology.orchestra.run.smithi138.stdout:Reading package lists... 2024-01-13T11:35:52.424 INFO:teuthology.orchestra.run.smithi088.stdout:The following additional packages will be installed: 2024-01-13T11:35:52.424 INFO:teuthology.orchestra.run.smithi088.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2024-01-13T11:35:52.424 INFO:teuthology.orchestra.run.smithi088.stdout: libusbmuxd6 linux-image-5.15.0-91-generic linux-modules-5.15.0-91-generic 2024-01-13T11:35:52.424 INFO:teuthology.orchestra.run.smithi088.stdout: linux-modules-extra-5.15.0-91-generic thermald upower usbmuxd 2024-01-13T11:35:52.425 INFO:teuthology.orchestra.run.smithi088.stdout:Suggested packages: 2024-01-13T11:35:52.425 INFO:teuthology.orchestra.run.smithi088.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.15-source-5.15.0 2024-01-13T11:35:52.425 INFO:teuthology.orchestra.run.smithi088.stdout: linux-hwe-5.15-tools linux-headers-5.15.0-91-generic 2024-01-13T11:35:52.461 INFO:teuthology.orchestra.run.smithi138.stdout:Building dependency tree... 2024-01-13T11:35:52.462 INFO:teuthology.orchestra.run.smithi138.stdout:Reading state information... 2024-01-13T11:35:52.483 INFO:teuthology.orchestra.run.smithi088.stdout:The following NEW packages will be installed: 2024-01-13T11:35:52.484 INFO:teuthology.orchestra.run.smithi088.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2024-01-13T11:35:52.484 INFO:teuthology.orchestra.run.smithi088.stdout: libusbmuxd6 linux-image-5.15.0-91-generic linux-image-generic-hwe-20.04 2024-01-13T11:35:52.484 INFO:teuthology.orchestra.run.smithi088.stdout: linux-modules-5.15.0-91-generic linux-modules-extra-5.15.0-91-generic 2024-01-13T11:35:52.485 INFO:teuthology.orchestra.run.smithi088.stdout: thermald upower usbmuxd 2024-01-13T11:35:52.546 INFO:teuthology.orchestra.run.smithi088.stdout:0 upgraded, 13 newly installed, 0 to remove and 285 not upgraded. 2024-01-13T11:35:52.546 INFO:teuthology.orchestra.run.smithi088.stdout:Need to get 90.9 MB of archives. 2024-01-13T11:35:52.546 INFO:teuthology.orchestra.run.smithi088.stdout:After this operation, 497 MB of additional disk space will be used. 2024-01-13T11:35:52.546 INFO:teuthology.orchestra.run.smithi088.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2024-01-13T11:35:52.612 INFO:teuthology.orchestra.run.smithi088.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2024-01-13T11:35:52.618 INFO:teuthology.orchestra.run.smithi088.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2024-01-13T11:35:52.623 INFO:teuthology.orchestra.run.smithi088.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2024-01-13T11:35:52.641 INFO:teuthology.orchestra.run.smithi088.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2024-01-13T11:35:52.642 INFO:teuthology.orchestra.run.smithi138.stdout:The following additional packages will be installed: 2024-01-13T11:35:52.643 INFO:teuthology.orchestra.run.smithi138.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2024-01-13T11:35:52.643 INFO:teuthology.orchestra.run.smithi138.stdout: libusbmuxd6 linux-image-5.15.0-91-generic linux-modules-5.15.0-91-generic 2024-01-13T11:35:52.643 INFO:teuthology.orchestra.run.smithi138.stdout: linux-modules-extra-5.15.0-91-generic thermald upower usbmuxd 2024-01-13T11:35:52.644 INFO:teuthology.orchestra.run.smithi138.stdout:Suggested packages: 2024-01-13T11:35:52.644 INFO:teuthology.orchestra.run.smithi138.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.15-source-5.15.0 2024-01-13T11:35:52.644 INFO:teuthology.orchestra.run.smithi138.stdout: linux-hwe-5.15-tools linux-headers-5.15.0-91-generic 2024-01-13T11:35:52.649 INFO:teuthology.orchestra.run.smithi088.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.15.0-91-generic amd64 5.15.0-91.101~20.04.1 [21.0 MB] 2024-01-13T11:35:52.705 INFO:teuthology.orchestra.run.smithi138.stdout:The following NEW packages will be installed: 2024-01-13T11:35:52.706 INFO:teuthology.orchestra.run.smithi138.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2024-01-13T11:35:52.706 INFO:teuthology.orchestra.run.smithi138.stdout: libusbmuxd6 linux-image-5.15.0-91-generic linux-image-generic-hwe-20.04 2024-01-13T11:35:52.706 INFO:teuthology.orchestra.run.smithi138.stdout: linux-modules-5.15.0-91-generic linux-modules-extra-5.15.0-91-generic 2024-01-13T11:35:52.706 INFO:teuthology.orchestra.run.smithi138.stdout: thermald upower usbmuxd 2024-01-13T11:35:52.888 INFO:teuthology.orchestra.run.smithi138.stdout:0 upgraded, 13 newly installed, 0 to remove and 285 not upgraded. 2024-01-13T11:35:52.888 INFO:teuthology.orchestra.run.smithi138.stdout:Need to get 90.9 MB of archives. 2024-01-13T11:35:52.888 INFO:teuthology.orchestra.run.smithi138.stdout:After this operation, 497 MB of additional disk space will be used. 2024-01-13T11:35:52.889 INFO:teuthology.orchestra.run.smithi138.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2024-01-13T11:35:53.030 INFO:teuthology.orchestra.run.smithi088.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.15.0-91-generic amd64 5.15.0-91.101~20.04.1 [11.4 MB] 2024-01-13T11:35:53.173 INFO:teuthology.orchestra.run.smithi138.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2024-01-13T11:35:53.178 INFO:teuthology.orchestra.run.smithi088.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.15.0-91-generic amd64 5.15.0-91.101~20.04.1 [57.9 MB] 2024-01-13T11:35:53.200 INFO:teuthology.orchestra.run.smithi138.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2024-01-13T11:35:53.218 INFO:teuthology.orchestra.run.smithi138.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2024-01-13T11:35:53.290 INFO:teuthology.orchestra.run.smithi138.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2024-01-13T11:35:53.325 INFO:teuthology.orchestra.run.smithi138.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.15.0-91-generic amd64 5.15.0-91.101~20.04.1 [21.0 MB] 2024-01-13T11:35:54.259 INFO:teuthology.orchestra.run.smithi138.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.15.0-91-generic amd64 5.15.0-91.101~20.04.1 [11.4 MB] 2024-01-13T11:35:54.303 INFO:teuthology.orchestra.run.smithi088.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.15.0.91.101~20.04.48 [2,628 B] 2024-01-13T11:35:54.303 INFO:teuthology.orchestra.run.smithi088.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.2 [31.6 kB] 2024-01-13T11:35:54.304 INFO:teuthology.orchestra.run.smithi088.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2024-01-13T11:35:54.310 INFO:teuthology.orchestra.run.smithi088.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2024-01-13T11:35:54.312 INFO:teuthology.orchestra.run.smithi088.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2024-01-13T11:35:54.545 INFO:teuthology.orchestra.run.smithi138.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.15.0-91-generic amd64 5.15.0-91.101~20.04.1 [57.9 MB] 2024-01-13T11:35:55.025 INFO:teuthology.orchestra.run.smithi088.stdout:Fetched 90.9 MB in 2s (49.9 MB/s) 2024-01-13T11:35:55.258 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2024-01-13T11:35:56.455 INFO:teuthology.orchestra.run.smithi138.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.15.0.91.101~20.04.48 [2,628 B] 2024-01-13T11:35:56.455 INFO:teuthology.orchestra.run.smithi138.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.2 [31.6 kB] 2024-01-13T11:35:56.456 INFO:teuthology.orchestra.run.smithi138.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2024-01-13T11:35:56.625 INFO:teuthology.orchestra.run.smithi138.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2024-01-13T11:35:56.971 INFO:teuthology.orchestra.run.smithi138.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2024-01-13T11:35:56.994 INFO:teuthology.orchestra.run.smithi088.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-01-13T11:35:56.999 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2024-01-13T11:35:57.214 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2024-01-13T11:35:57.541 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libplist3:amd64. 2024-01-13T11:35:57.556 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2024-01-13T11:35:57.610 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2024-01-13T11:35:57.651 INFO:teuthology.orchestra.run.smithi138.stdout:Fetched 90.9 MB in 4s (21.2 MB/s) 2024-01-13T11:35:57.882 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2024-01-13T11:35:57.944 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2024-01-13T11:35:57.958 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2024-01-13T11:35:57.986 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2024-01-13T11:35:58.347 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2024-01-13T11:35:58.361 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2024-01-13T11:35:58.406 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2024-01-13T11:35:58.800 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libupower-glib3:amd64. 2024-01-13T11:35:58.813 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2024-01-13T11:35:58.858 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2024-01-13T11:35:59.170 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package linux-modules-5.15.0-91-generic. 2024-01-13T11:35:59.182 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../05-linux-modules-5.15.0-91-generic_5.15.0-91.101~20.04.1_amd64.deb ... 2024-01-13T11:35:59.220 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking linux-modules-5.15.0-91-generic (5.15.0-91.101~20.04.1) ... 2024-01-13T11:35:59.661 INFO:teuthology.orchestra.run.smithi138.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-01-13T11:35:59.666 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2024-01-13T11:35:59.848 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2024-01-13T11:36:00.184 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libplist3:amd64. 2024-01-13T11:36:00.200 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2024-01-13T11:36:00.268 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2024-01-13T11:36:00.646 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2024-01-13T11:36:00.659 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2024-01-13T11:36:00.696 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2024-01-13T11:36:01.050 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2024-01-13T11:36:01.066 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2024-01-13T11:36:01.108 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2024-01-13T11:36:01.462 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libupower-glib3:amd64. 2024-01-13T11:36:01.478 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2024-01-13T11:36:01.537 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2024-01-13T11:36:01.796 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package linux-image-5.15.0-91-generic. 2024-01-13T11:36:01.810 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../06-linux-image-5.15.0-91-generic_5.15.0-91.101~20.04.1_amd64.deb ... 2024-01-13T11:36:01.874 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package linux-modules-5.15.0-91-generic. 2024-01-13T11:36:01.888 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../05-linux-modules-5.15.0-91-generic_5.15.0-91.101~20.04.1_amd64.deb ... 2024-01-13T11:36:01.918 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking linux-image-5.15.0-91-generic (5.15.0-91.101~20.04.1) ... 2024-01-13T11:36:01.932 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking linux-modules-5.15.0-91-generic (5.15.0-91.101~20.04.1) ... 2024-01-13T11:36:02.416 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-91-generic. 2024-01-13T11:36:02.428 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../07-linux-modules-extra-5.15.0-91-generic_5.15.0-91.101~20.04.1_amd64.deb ... 2024-01-13T11:36:02.482 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking linux-modules-extra-5.15.0-91-generic (5.15.0-91.101~20.04.1) ... 2024-01-13T11:36:04.486 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package linux-image-5.15.0-91-generic. 2024-01-13T11:36:04.502 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../06-linux-image-5.15.0-91-generic_5.15.0-91.101~20.04.1_amd64.deb ... 2024-01-13T11:36:04.593 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking linux-image-5.15.0-91-generic (5.15.0-91.101~20.04.1) ... 2024-01-13T11:36:05.132 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-91-generic. 2024-01-13T11:36:05.145 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../07-linux-modules-extra-5.15.0-91-generic_5.15.0-91.101~20.04.1_amd64.deb ... 2024-01-13T11:36:05.198 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking linux-modules-extra-5.15.0-91-generic (5.15.0-91.101~20.04.1) ... 2024-01-13T11:36:09.609 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2024-01-13T11:36:09.624 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.15.0.91.101~20.04.48_amd64.deb ... 2024-01-13T11:36:09.667 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking linux-image-generic-hwe-20.04 (5.15.0.91.101~20.04.48) ... 2024-01-13T11:36:10.003 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libevdev2:amd64. 2024-01-13T11:36:10.019 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.2_amd64.deb ... 2024-01-13T11:36:10.069 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.2) ... 2024-01-13T11:36:10.397 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package thermald. 2024-01-13T11:36:10.411 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2024-01-13T11:36:10.439 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2024-01-13T11:36:11.030 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package upower. 2024-01-13T11:36:11.047 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2024-01-13T11:36:11.105 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking upower (0.99.11-1build2) ... 2024-01-13T11:36:11.534 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package usbmuxd. 2024-01-13T11:36:11.547 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2024-01-13T11:36:11.592 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2024-01-13T11:36:12.327 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2024-01-13T11:36:12.495 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2024-01-13T11:36:12.510 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.15.0.91.101~20.04.48_amd64.deb ... 2024-01-13T11:36:12.568 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking linux-image-generic-hwe-20.04 (5.15.0.91.101~20.04.48) ... 2024-01-13T11:36:12.596 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2024-01-13T11:36:12.722 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2024-01-13T11:36:12.873 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libevdev2:amd64. 2024-01-13T11:36:12.873 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2024-01-13T11:36:12.886 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.2_amd64.deb ... 2024-01-13T11:36:12.914 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.2) ... 2024-01-13T11:36:12.991 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2024-01-13T11:36:13.142 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.2) ... 2024-01-13T11:36:13.293 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up upower (0.99.11-1build2) ... 2024-01-13T11:36:13.310 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package thermald. 2024-01-13T11:36:13.324 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2024-01-13T11:36:13.368 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2024-01-13T11:36:13.831 INFO:teuthology.orchestra.run.smithi088.stdout:upower.service is a disabled or a static unit, not starting it. 2024-01-13T11:36:13.889 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2024-01-13T11:36:13.940 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package upower. 2024-01-13T11:36:13.957 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2024-01-13T11:36:14.053 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking upower (0.99.11-1build2) ... 2024-01-13T11:36:14.086 INFO:teuthology.orchestra.run.smithi088.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2024-01-13T11:36:14.097 INFO:teuthology.orchestra.run.smithi088.stdout:Adding system user `usbmux' (UID 114) ... 2024-01-13T11:36:14.097 INFO:teuthology.orchestra.run.smithi088.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2024-01-13T11:36:14.504 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package usbmuxd. 2024-01-13T11:36:14.520 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2024-01-13T11:36:14.562 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2024-01-13T11:36:14.589 INFO:teuthology.orchestra.run.smithi088.stdout:Not creating home directory `/var/lib/usbmux'. 2024-01-13T11:36:14.809 INFO:teuthology.orchestra.run.smithi088.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2024-01-13T11:36:14.861 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2024-01-13T11:36:14.875 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2024-01-13T11:36:15.001 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2024-01-13T11:36:15.127 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2024-01-13T11:36:15.191 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2024-01-13T11:36:15.191 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2024-01-13T11:36:15.279 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2024-01-13T11:36:15.413 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2024-01-13T11:36:15.540 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.2) ... 2024-01-13T11:36:15.641 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up linux-image-5.15.0-91-generic (5.15.0-91.101~20.04.1) ... 2024-01-13T11:36:15.691 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up upower (0.99.11-1build2) ... 2024-01-13T11:36:16.248 INFO:teuthology.orchestra.run.smithi138.stdout:upower.service is a disabled or a static unit, not starting it. 2024-01-13T11:36:16.304 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2024-01-13T11:36:16.486 INFO:teuthology.orchestra.run.smithi138.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2024-01-13T11:36:16.498 INFO:teuthology.orchestra.run.smithi138.stdout:Adding system user `usbmux' (UID 114) ... 2024-01-13T11:36:16.498 INFO:teuthology.orchestra.run.smithi138.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2024-01-13T11:36:16.947 INFO:teuthology.orchestra.run.smithi088.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-91-generic 2024-01-13T11:36:16.948 INFO:teuthology.orchestra.run.smithi088.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-91-generic 2024-01-13T11:36:17.031 INFO:teuthology.orchestra.run.smithi138.stdout:Not creating home directory `/var/lib/usbmux'. 2024-01-13T11:36:17.125 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up linux-modules-extra-5.15.0-91-generic (5.15.0-91.101~20.04.1) ... 2024-01-13T11:36:17.242 INFO:teuthology.orchestra.run.smithi138.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2024-01-13T11:36:17.285 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2024-01-13T11:36:17.972 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2024-01-13T11:36:17.972 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2024-01-13T11:36:18.608 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up linux-image-5.15.0-91-generic (5.15.0-91.101~20.04.1) ... 2024-01-13T11:36:18.901 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up linux-image-generic-hwe-20.04 (5.15.0.91.101~20.04.48) ... 2024-01-13T11:36:19.044 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up linux-modules-5.15.0-91-generic (5.15.0-91.101~20.04.1) ... 2024-01-13T11:36:19.931 INFO:teuthology.orchestra.run.smithi138.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-91-generic 2024-01-13T11:36:19.932 INFO:teuthology.orchestra.run.smithi138.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-91-generic 2024-01-13T11:36:20.120 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up linux-modules-extra-5.15.0-91-generic (5.15.0-91.101~20.04.1) ... 2024-01-13T11:36:20.437 INFO:teuthology.orchestra.run.smithi088.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-01-13T11:36:20.742 INFO:teuthology.orchestra.run.smithi088.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-01-13T11:36:21.514 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up linux-image-generic-hwe-20.04 (5.15.0.91.101~20.04.48) ... 2024-01-13T11:36:21.537 INFO:teuthology.orchestra.run.smithi088.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.2) ... 2024-01-13T11:36:21.639 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up linux-modules-5.15.0-91-generic (5.15.0-91.101~20.04.1) ... 2024-01-13T11:36:21.662 INFO:teuthology.orchestra.run.smithi088.stdout:Processing triggers for linux-image-5.15.0-91-generic (5.15.0-91.101~20.04.1) ... 2024-01-13T11:36:21.736 INFO:teuthology.orchestra.run.smithi088.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-01-13T11:36:21.736 INFO:teuthology.orchestra.run.smithi088.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-91-generic 2024-01-13T11:36:23.025 INFO:teuthology.orchestra.run.smithi138.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-01-13T11:36:23.306 INFO:teuthology.orchestra.run.smithi138.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-01-13T11:36:24.826 INFO:teuthology.orchestra.run.smithi138.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.2) ... 2024-01-13T11:36:24.960 INFO:teuthology.orchestra.run.smithi138.stdout:Processing triggers for linux-image-5.15.0-91-generic (5.15.0-91.101~20.04.1) ... 2024-01-13T11:36:25.030 INFO:teuthology.orchestra.run.smithi138.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-01-13T11:36:25.030 INFO:teuthology.orchestra.run.smithi138.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-91-generic 2024-01-13T11:36:47.088 INFO:teuthology.orchestra.run.smithi088.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-01-13T11:36:47.089 INFO:teuthology.orchestra.run.smithi088.stdout:Sourcing file `/etc/default/grub' 2024-01-13T11:36:47.100 INFO:teuthology.orchestra.run.smithi088.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-01-13T11:36:47.152 INFO:teuthology.orchestra.run.smithi088.stdout:Generating grub configuration file ... 2024-01-13T11:36:47.498 INFO:teuthology.orchestra.run.smithi088.stdout:Found linux image: /boot/vmlinuz-5.15.0-91-generic 2024-01-13T11:36:47.519 INFO:teuthology.orchestra.run.smithi088.stdout:Found initrd image: /boot/initrd.img-5.15.0-91-generic 2024-01-13T11:36:48.041 INFO:teuthology.orchestra.run.smithi088.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-01-13T11:36:48.055 INFO:teuthology.orchestra.run.smithi088.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-01-13T11:36:50.571 INFO:teuthology.orchestra.run.smithi138.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-01-13T11:36:50.571 INFO:teuthology.orchestra.run.smithi138.stdout:Sourcing file `/etc/default/grub' 2024-01-13T11:36:50.578 INFO:teuthology.orchestra.run.smithi138.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-01-13T11:36:50.605 INFO:teuthology.orchestra.run.smithi138.stdout:Generating grub configuration file ... 2024-01-13T11:36:50.803 INFO:teuthology.orchestra.run.smithi088.stdout:done 2024-01-13T11:36:50.972 INFO:teuthology.orchestra.run.smithi138.stdout:Found linux image: /boot/vmlinuz-5.15.0-91-generic 2024-01-13T11:36:50.996 INFO:teuthology.orchestra.run.smithi138.stdout:Found initrd image: /boot/initrd.img-5.15.0-91-generic 2024-01-13T11:36:51.280 DEBUG:teuthology.orchestra.run.smithi088:> dpkg -s linux-image-generic-hwe-20.04 2024-01-13T11:36:51.329 INFO:teuthology.orchestra.run.smithi088.stdout:Package: linux-image-generic-hwe-20.04 2024-01-13T11:36:51.329 INFO:teuthology.orchestra.run.smithi088.stdout:Status: install ok installed 2024-01-13T11:36:51.329 INFO:teuthology.orchestra.run.smithi088.stdout:Priority: optional 2024-01-13T11:36:51.329 INFO:teuthology.orchestra.run.smithi088.stdout:Section: kernel 2024-01-13T11:36:51.329 INFO:teuthology.orchestra.run.smithi088.stdout:Installed-Size: 20 2024-01-13T11:36:51.329 INFO:teuthology.orchestra.run.smithi088.stdout:Maintainer: Ubuntu Kernel Team 2024-01-13T11:36:51.329 INFO:teuthology.orchestra.run.smithi088.stdout:Architecture: amd64 2024-01-13T11:36:51.329 INFO:teuthology.orchestra.run.smithi088.stdout:Source: linux-meta-hwe-5.15 2024-01-13T11:36:51.330 INFO:teuthology.orchestra.run.smithi088.stdout:Version: 5.15.0.91.101~20.04.48 2024-01-13T11:36:51.330 INFO:teuthology.orchestra.run.smithi088.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-91), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2024-01-13T11:36:51.330 INFO:teuthology.orchestra.run.smithi088.stdout:Depends: linux-image-5.15.0-91-generic, linux-modules-extra-5.15.0-91-generic, linux-firmware, intel-microcode, amd64-microcode 2024-01-13T11:36:51.330 INFO:teuthology.orchestra.run.smithi088.stdout:Recommends: thermald 2024-01-13T11:36:51.330 INFO:teuthology.orchestra.run.smithi088.stdout:Description: Generic Linux kernel image 2024-01-13T11:36:51.330 INFO:teuthology.orchestra.run.smithi088.stdout: This package will always depend on the latest generic kernel image 2024-01-13T11:36:51.330 INFO:teuthology.orchestra.run.smithi088.stdout: available. 2024-01-13T11:36:51.330 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.15.0-91-generic 2024-01-13T11:36:51.330 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-13T11:36:51.331 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-13T11:36:51.331 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-01-13T11:36:51.331 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi088.front.sepia.ceph.com, path=None, version=distro) 2024-01-13T11:36:51.331 DEBUG:teuthology.orchestra.run.smithi088:> sudo apt-get clean 2024-01-13T11:36:51.404 DEBUG:teuthology.orchestra.run.smithi088:> sudo apt-get update 2024-01-13T11:36:51.473 INFO:teuthology.orchestra.run.smithi138.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-01-13T11:36:51.485 INFO:teuthology.orchestra.run.smithi138.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-01-13T11:36:51.630 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-01-13T11:36:51.630 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-01-13T11:36:51.644 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-01-13T11:36:51.668 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-01-13T11:36:53.393 INFO:teuthology.orchestra.run.smithi088.stdout:Reading package lists... 2024-01-13T11:36:53.421 DEBUG:teuthology.orchestra.run.smithi088:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2024-01-13T11:36:53.489 INFO:teuthology.orchestra.run.smithi088.stdout:Reading package lists... 2024-01-13T11:36:53.703 INFO:teuthology.orchestra.run.smithi088.stdout:Building dependency tree... 2024-01-13T11:36:53.704 INFO:teuthology.orchestra.run.smithi088.stdout:Reading state information... 2024-01-13T11:36:53.923 INFO:teuthology.orchestra.run.smithi088.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.15.0.91.101~20.04.48). 2024-01-13T11:36:53.924 INFO:teuthology.orchestra.run.smithi088.stdout:0 upgraded, 0 newly installed, 0 to remove and 285 not upgraded. 2024-01-13T11:36:53.926 DEBUG:teuthology.orchestra.run.smithi088:> dpkg -s linux-image-generic-hwe-20.04 2024-01-13T11:36:53.945 INFO:teuthology.orchestra.run.smithi088.stdout:Package: linux-image-generic-hwe-20.04 2024-01-13T11:36:53.945 INFO:teuthology.orchestra.run.smithi088.stdout:Status: install ok installed 2024-01-13T11:36:53.945 INFO:teuthology.orchestra.run.smithi088.stdout:Priority: optional 2024-01-13T11:36:53.945 INFO:teuthology.orchestra.run.smithi088.stdout:Section: kernel 2024-01-13T11:36:53.945 INFO:teuthology.orchestra.run.smithi088.stdout:Installed-Size: 20 2024-01-13T11:36:53.945 INFO:teuthology.orchestra.run.smithi088.stdout:Maintainer: Ubuntu Kernel Team 2024-01-13T11:36:53.945 INFO:teuthology.orchestra.run.smithi088.stdout:Architecture: amd64 2024-01-13T11:36:53.946 INFO:teuthology.orchestra.run.smithi088.stdout:Source: linux-meta-hwe-5.15 2024-01-13T11:36:53.946 INFO:teuthology.orchestra.run.smithi088.stdout:Version: 5.15.0.91.101~20.04.48 2024-01-13T11:36:53.946 INFO:teuthology.orchestra.run.smithi088.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-91), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2024-01-13T11:36:53.946 INFO:teuthology.orchestra.run.smithi088.stdout:Depends: linux-image-5.15.0-91-generic, linux-modules-extra-5.15.0-91-generic, linux-firmware, intel-microcode, amd64-microcode 2024-01-13T11:36:53.946 INFO:teuthology.orchestra.run.smithi088.stdout:Recommends: thermald 2024-01-13T11:36:53.946 INFO:teuthology.orchestra.run.smithi088.stdout:Description: Generic Linux kernel image 2024-01-13T11:36:53.946 INFO:teuthology.orchestra.run.smithi088.stdout: This package will always depend on the latest generic kernel image 2024-01-13T11:36:53.946 INFO:teuthology.orchestra.run.smithi088.stdout: available. 2024-01-13T11:36:53.946 DEBUG:teuthology.orchestra.run.smithi088:> mktemp 2024-01-13T11:36:53.950 INFO:teuthology.orchestra.run.smithi088.stdout:/tmp/tmp.oq8vNnotmD 2024-01-13T11:36:53.951 DEBUG:teuthology.orchestra.run.smithi088:> sudo cp /boot/grub/grub.cfg /tmp/tmp.oq8vNnotmD 2024-01-13T11:36:53.960 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0666 /tmp/tmp.oq8vNnotmD 2024-01-13T11:36:54.000 DEBUG:teuthology.orchestra.remote:smithi088:/tmp/tmp.oq8vNnotmD is 9KB 2024-01-13T11:36:54.014 DEBUG:teuthology.orchestra.run.smithi088:> rm -fr /tmp/tmp.oq8vNnotmD 2024-01-13T11:36:54.021 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-01-13T11:36:54.076 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-01-13T11:36:54.076 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-01-13T11:36:54.076 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-01-13T11:36:54.131 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-91-generic 2024-01-13T11:36:54.131 DEBUG:teuthology.orchestra.run.smithi088:> sudo update-grub 2024-01-13T11:36:54.175 INFO:teuthology.orchestra.run.smithi138.stdout:done 2024-01-13T11:36:54.662 DEBUG:teuthology.orchestra.run.smithi138:> dpkg -s linux-image-generic-hwe-20.04 2024-01-13T11:36:54.715 INFO:teuthology.orchestra.run.smithi138.stdout:Package: linux-image-generic-hwe-20.04 2024-01-13T11:36:54.715 INFO:teuthology.orchestra.run.smithi138.stdout:Status: install ok installed 2024-01-13T11:36:54.715 INFO:teuthology.orchestra.run.smithi138.stdout:Priority: optional 2024-01-13T11:36:54.715 INFO:teuthology.orchestra.run.smithi138.stdout:Section: kernel 2024-01-13T11:36:54.715 INFO:teuthology.orchestra.run.smithi138.stdout:Installed-Size: 20 2024-01-13T11:36:54.715 INFO:teuthology.orchestra.run.smithi138.stdout:Maintainer: Ubuntu Kernel Team 2024-01-13T11:36:54.715 INFO:teuthology.orchestra.run.smithi138.stdout:Architecture: amd64 2024-01-13T11:36:54.715 INFO:teuthology.orchestra.run.smithi138.stdout:Source: linux-meta-hwe-5.15 2024-01-13T11:36:54.715 INFO:teuthology.orchestra.run.smithi138.stdout:Version: 5.15.0.91.101~20.04.48 2024-01-13T11:36:54.715 INFO:teuthology.orchestra.run.smithi138.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-91), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2024-01-13T11:36:54.715 INFO:teuthology.orchestra.run.smithi138.stdout:Depends: linux-image-5.15.0-91-generic, linux-modules-extra-5.15.0-91-generic, linux-firmware, intel-microcode, amd64-microcode 2024-01-13T11:36:54.716 INFO:teuthology.orchestra.run.smithi138.stdout:Recommends: thermald 2024-01-13T11:36:54.716 INFO:teuthology.orchestra.run.smithi138.stdout:Description: Generic Linux kernel image 2024-01-13T11:36:54.716 INFO:teuthology.orchestra.run.smithi138.stdout: This package will always depend on the latest generic kernel image 2024-01-13T11:36:54.716 INFO:teuthology.orchestra.run.smithi138.stdout: available. 2024-01-13T11:36:54.716 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.15.0-91-generic 2024-01-13T11:36:54.716 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-13T11:36:54.716 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-13T11:36:54.716 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-01-13T11:36:54.716 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi138.front.sepia.ceph.com, path=None, version=distro) 2024-01-13T11:36:54.717 DEBUG:teuthology.orchestra.run.smithi138:> sudo apt-get clean 2024-01-13T11:36:54.789 DEBUG:teuthology.orchestra.run.smithi138:> sudo apt-get update 2024-01-13T11:36:55.058 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-01-13T11:36:55.058 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-01-13T11:36:55.129 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-01-13T11:36:55.211 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-01-13T11:36:55.595 INFO:teuthology.orchestra.run.smithi088.stderr:Sourcing file `/etc/default/grub' 2024-01-13T11:36:55.605 INFO:teuthology.orchestra.run.smithi088.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-01-13T11:36:55.650 INFO:teuthology.orchestra.run.smithi088.stderr:Generating grub configuration file ... 2024-01-13T11:36:55.992 INFO:teuthology.orchestra.run.smithi088.stderr:Found linux image: /boot/vmlinuz-5.15.0-91-generic 2024-01-13T11:36:56.004 INFO:teuthology.orchestra.run.smithi088.stderr:Found initrd image: /boot/initrd.img-5.15.0-91-generic 2024-01-13T11:36:56.466 INFO:teuthology.orchestra.run.smithi088.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-01-13T11:36:56.477 INFO:teuthology.orchestra.run.smithi088.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-01-13T11:36:56.819 INFO:teuthology.orchestra.run.smithi138.stdout:Reading package lists... 2024-01-13T11:36:56.842 DEBUG:teuthology.orchestra.run.smithi138:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2024-01-13T11:36:56.913 INFO:teuthology.orchestra.run.smithi138.stdout:Reading package lists... 2024-01-13T11:36:57.119 INFO:teuthology.orchestra.run.smithi138.stdout:Building dependency tree... 2024-01-13T11:36:57.119 INFO:teuthology.orchestra.run.smithi138.stdout:Reading state information... 2024-01-13T11:36:57.341 INFO:teuthology.orchestra.run.smithi138.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.15.0.91.101~20.04.48). 2024-01-13T11:36:57.341 INFO:teuthology.orchestra.run.smithi138.stdout:0 upgraded, 0 newly installed, 0 to remove and 285 not upgraded. 2024-01-13T11:36:57.342 DEBUG:teuthology.orchestra.run.smithi138:> dpkg -s linux-image-generic-hwe-20.04 2024-01-13T11:36:57.362 INFO:teuthology.orchestra.run.smithi138.stdout:Package: linux-image-generic-hwe-20.04 2024-01-13T11:36:57.363 INFO:teuthology.orchestra.run.smithi138.stdout:Status: install ok installed 2024-01-13T11:36:57.363 INFO:teuthology.orchestra.run.smithi138.stdout:Priority: optional 2024-01-13T11:36:57.363 INFO:teuthology.orchestra.run.smithi138.stdout:Section: kernel 2024-01-13T11:36:57.363 INFO:teuthology.orchestra.run.smithi138.stdout:Installed-Size: 20 2024-01-13T11:36:57.363 INFO:teuthology.orchestra.run.smithi138.stdout:Maintainer: Ubuntu Kernel Team 2024-01-13T11:36:57.363 INFO:teuthology.orchestra.run.smithi138.stdout:Architecture: amd64 2024-01-13T11:36:57.363 INFO:teuthology.orchestra.run.smithi138.stdout:Source: linux-meta-hwe-5.15 2024-01-13T11:36:57.363 INFO:teuthology.orchestra.run.smithi138.stdout:Version: 5.15.0.91.101~20.04.48 2024-01-13T11:36:57.363 INFO:teuthology.orchestra.run.smithi138.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-91), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2024-01-13T11:36:57.363 INFO:teuthology.orchestra.run.smithi138.stdout:Depends: linux-image-5.15.0-91-generic, linux-modules-extra-5.15.0-91-generic, linux-firmware, intel-microcode, amd64-microcode 2024-01-13T11:36:57.363 INFO:teuthology.orchestra.run.smithi138.stdout:Recommends: thermald 2024-01-13T11:36:57.363 INFO:teuthology.orchestra.run.smithi138.stdout:Description: Generic Linux kernel image 2024-01-13T11:36:57.363 INFO:teuthology.orchestra.run.smithi138.stdout: This package will always depend on the latest generic kernel image 2024-01-13T11:36:57.363 INFO:teuthology.orchestra.run.smithi138.stdout: available. 2024-01-13T11:36:57.364 DEBUG:teuthology.orchestra.run.smithi138:> mktemp 2024-01-13T11:36:57.368 INFO:teuthology.orchestra.run.smithi138.stdout:/tmp/tmp.x0LInuMs2c 2024-01-13T11:36:57.368 DEBUG:teuthology.orchestra.run.smithi138:> sudo cp /boot/grub/grub.cfg /tmp/tmp.x0LInuMs2c 2024-01-13T11:36:57.424 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 0666 /tmp/tmp.x0LInuMs2c 2024-01-13T11:36:57.466 DEBUG:teuthology.orchestra.remote:smithi138:/tmp/tmp.x0LInuMs2c is 9KB 2024-01-13T11:36:57.480 DEBUG:teuthology.orchestra.run.smithi138:> rm -fr /tmp/tmp.x0LInuMs2c 2024-01-13T11:36:57.486 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-01-13T11:36:57.541 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-01-13T11:36:57.541 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-01-13T11:36:57.541 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-01-13T11:36:57.595 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-91-generic 2024-01-13T11:36:57.595 DEBUG:teuthology.orchestra.run.smithi138:> sudo update-grub 2024-01-13T11:36:58.143 INFO:teuthology.orchestra.run.smithi088.stderr:done 2024-01-13T11:36:58.145 DEBUG:teuthology.orchestra.run.smithi088:> sudo shutdown -r now 2024-01-13T11:36:59.071 INFO:teuthology.orchestra.run.smithi138.stderr:Sourcing file `/etc/default/grub' 2024-01-13T11:36:59.076 INFO:teuthology.orchestra.run.smithi138.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-01-13T11:36:59.087 INFO:teuthology.orchestra.run.smithi138.stderr:Generating grub configuration file ... 2024-01-13T11:36:59.392 INFO:teuthology.orchestra.run.smithi138.stderr:Found linux image: /boot/vmlinuz-5.15.0-91-generic 2024-01-13T11:36:59.403 INFO:teuthology.orchestra.run.smithi138.stderr:Found initrd image: /boot/initrd.img-5.15.0-91-generic 2024-01-13T11:36:59.854 INFO:teuthology.orchestra.run.smithi138.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-01-13T11:36:59.865 INFO:teuthology.orchestra.run.smithi138.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-01-13T11:37:01.531 INFO:teuthology.orchestra.run.smithi138.stderr:done 2024-01-13T11:37:01.533 DEBUG:teuthology.orchestra.run.smithi138:> sudo shutdown -r now 2024-01-13T11:37:28.172 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-01-13T11:37:28.172 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-01-13T11:37:28.173 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-13T11:37:31.539 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-01-13T11:37:31.540 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-01-13T11:37:31.540 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-13T11:37:46.556 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-01-13T11:37:55.564 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-01-13T11:37:55.565 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-13T11:37:58.620 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-01-13T11:38:06.012 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.138 2024-01-13T11:38:10.624 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-01-13T11:38:10.624 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-13T11:38:13.692 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-01-13T11:38:15.014 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-01-13T11:38:15.014 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-13T11:38:28.704 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-01-13T11:38:28.704 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-13T11:38:33.404 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.138 2024-01-13T11:38:45.412 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-01-13T11:38:45.413 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-13T11:38:46.077 DEBUG:teuthology.orchestra.run.smithi138:> true 2024-01-13T11:38:46.742 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-01-13T11:38:46.742 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.15.0-91-generic"... 2024-01-13T11:38:46.742 DEBUG:teuthology.orchestra.run.smithi138:> uname -r 2024-01-13T11:38:46.788 INFO:teuthology.orchestra.run.smithi138.stdout:5.15.0-91-generic 2024-01-13T11:38:46.788 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-91-generic vs 5.15.0-91-generic 2024-01-13T11:38:46.788 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-13T11:38:46.788 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-01-13T11:38:47.228 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-01-13T11:38:47.789 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-01-13T11:38:47.789 DEBUG:teuthology.orchestra.run.smithi138:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-13T11:38:47.939 INFO:teuthology.orchestra.run.smithi138.stdout:ttyS1 2024-01-13T11:38:47.966 DEBUG:teuthology.parallel:result is None 2024-01-13T11:39:05.229 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-01-13T11:39:05.230 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-13T11:39:05.536 DEBUG:teuthology.orchestra.run.smithi088:> true 2024-01-13T11:39:06.085 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-01-13T11:39:06.085 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.15.0-91-generic"... 2024-01-13T11:39:06.085 DEBUG:teuthology.orchestra.run.smithi088:> uname -r 2024-01-13T11:39:06.133 INFO:teuthology.orchestra.run.smithi088.stdout:5.15.0-91-generic 2024-01-13T11:39:06.133 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-91-generic vs 5.15.0-91-generic 2024-01-13T11:39:06.133 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-13T11:39:06.133 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-01-13T11:39:07.134 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-01-13T11:39:07.135 DEBUG:teuthology.orchestra.run.smithi088:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-13T11:39:07.487 INFO:teuthology.orchestra.run.smithi088.stdout:ttyS1 2024-01-13T11:39:07.513 DEBUG:teuthology.parallel:result is None 2024-01-13T11:39:07.513 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-13T11:39:07.520 INFO:teuthology.task.internal:Creating test directory... 2024-01-13T11:39:07.521 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-13T11:39:07.523 DEBUG:teuthology.orchestra.run.smithi138:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-13T11:39:07.531 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-13T11:39:07.537 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-13T11:39:07.544 INFO:teuthology.task.internal:Creating archive directory... 2024-01-13T11:39:07.544 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-13T11:39:07.572 DEBUG:teuthology.orchestra.run.smithi138:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-13T11:39:07.639 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-13T11:39:07.646 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-13T11:39:07.646 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-13T11:39:07.648 DEBUG:teuthology.orchestra.run.smithi138:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-13T11:39:07.668 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-13T11:39:07.670 INFO:teuthology.orchestra.run.smithi138.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-13T11:39:07.674 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-13T11:39:07.679 INFO:teuthology.orchestra.run.smithi138.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-13T11:39:07.681 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-13T11:39:07.688 INFO:teuthology.task.internal:Configuring sudo... 2024-01-13T11:39:07.688 DEBUG:teuthology.orchestra.run.smithi088:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-13T11:39:07.720 DEBUG:teuthology.orchestra.run.smithi138:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-13T11:39:07.739 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-13T11:39:07.764 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-13T11:39:07.764 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-13T11:39:07.780 DEBUG:teuthology.orchestra.run.smithi138:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-13T11:39:07.790 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-13T11:39:07.833 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-13T11:39:07.881 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-01-13T11:39:07.881 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-13T11:39:07.947 DEBUG:teuthology.orchestra.run.smithi138:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-13T11:39:07.954 DEBUG:teuthology.orchestra.run.smithi138:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-13T11:39:08.002 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-01-13T11:39:08.002 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-13T11:39:08.070 DEBUG:teuthology.orchestra.run.smithi088:> sudo service rsyslog restart 2024-01-13T11:39:08.072 DEBUG:teuthology.orchestra.run.smithi138:> sudo service rsyslog restart 2024-01-13T11:39:08.166 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-13T11:39:08.173 INFO:teuthology.task.internal:Starting timer... 2024-01-13T11:39:08.173 INFO:teuthology.run_tasks:Running task pcp... 2024-01-13T11:39:08.219 INFO:teuthology.run_tasks:Running task selinux... 2024-01-13T11:39:08.270 DEBUG:teuthology.task.selinux:Excluding smithi088: OS 'ubuntu' does not support SELinux 2024-01-13T11:39:08.271 DEBUG:teuthology.task.selinux:Excluding smithi138: OS 'ubuntu' does not support SELinux 2024-01-13T11:39:08.271 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-13T11:39:08.271 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-01-13T11:39:08.271 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-13T11:39:08.271 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-13T11:39:08.285 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-13T11:39:08.301 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-13T11:39:08.303 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi088.front.sepia.ceph.com,smithi138.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-13T11:44:29.993 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi088.front.sepia.ceph.com'), Remote(name='ubuntu@smithi138.front.sepia.ceph.com')] 2024-01-13T11:44:29.995 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-01-13T11:44:29.996 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-13T11:44:30.085 DEBUG:teuthology.orchestra.run.smithi088:> true 2024-01-13T11:44:30.205 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-01-13T11:44:30.206 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-01-13T11:44:30.206 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-13T11:44:30.298 DEBUG:teuthology.orchestra.run.smithi138:> true 2024-01-13T11:44:30.425 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-01-13T11:44:30.426 INFO:teuthology.run_tasks:Running task clock... 2024-01-13T11:44:30.436 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-13T11:44:30.436 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-13T11:44:30.436 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-13T11:44:30.439 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-13T11:44:30.439 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-13T11:44:30.469 INFO:teuthology.orchestra.run.smithi088.stdout:13 Jan 11:44:30 ntpd[12577]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-01-13T11:44:30.469 INFO:teuthology.orchestra.run.smithi088.stdout:13 Jan 11:44:30 ntpd[12577]: Command line: ntpd -gq 2024-01-13T11:44:30.469 INFO:teuthology.orchestra.run.smithi088.stdout:13 Jan 11:44:30 ntpd[12577]: proto: precision = 0.072 usec (-24) 2024-01-13T11:44:30.470 INFO:teuthology.orchestra.run.smithi088.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-01-13T11:44:30.470 INFO:teuthology.orchestra.run.smithi088.stdout:13 Jan 11:44:30 ntpd[12577]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-01-13T11:44:30.470 INFO:teuthology.orchestra.run.smithi088.stderr:restrict ::: KOD does nothing without LIMITED. 2024-01-13T11:44:30.471 INFO:teuthology.orchestra.run.smithi088.stdout:13 Jan 11:44:30 ntpd[12577]: restrict ::: KOD does nothing without LIMITED. 2024-01-13T11:44:30.471 INFO:teuthology.orchestra.run.smithi088.stdout:13 Jan 11:44:30 ntpd[12577]: Listen and drop on 0 v6wildcard [::]:123 2024-01-13T11:44:30.472 INFO:teuthology.orchestra.run.smithi088.stdout:13 Jan 11:44:30 ntpd[12577]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-01-13T11:44:30.472 INFO:teuthology.orchestra.run.smithi088.stdout:13 Jan 11:44:30 ntpd[12577]: Listen normally on 2 lo 127.0.0.1:123 2024-01-13T11:44:30.472 INFO:teuthology.orchestra.run.smithi088.stdout:13 Jan 11:44:30 ntpd[12577]: Listen normally on 3 enp3s0f1 172.21.15.88:123 2024-01-13T11:44:30.472 INFO:teuthology.orchestra.run.smithi088.stdout:13 Jan 11:44:30 ntpd[12577]: Listen normally on 4 lo [::1]:123 2024-01-13T11:44:30.472 INFO:teuthology.orchestra.run.smithi088.stdout:13 Jan 11:44:30 ntpd[12577]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d42d%5]:123 2024-01-13T11:44:30.472 INFO:teuthology.orchestra.run.smithi088.stdout:13 Jan 11:44:30 ntpd[12577]: Listening on routing socket on fd #22 for interface updates 2024-01-13T11:44:30.498 INFO:teuthology.orchestra.run.smithi138.stdout:13 Jan 11:44:30 ntpd[12563]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-01-13T11:44:30.498 INFO:teuthology.orchestra.run.smithi138.stdout:13 Jan 11:44:30 ntpd[12563]: Command line: ntpd -gq 2024-01-13T11:44:30.498 INFO:teuthology.orchestra.run.smithi138.stdout:13 Jan 11:44:30 ntpd[12563]: proto: precision = 0.070 usec (-24) 2024-01-13T11:44:30.499 INFO:teuthology.orchestra.run.smithi138.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-01-13T11:44:30.499 INFO:teuthology.orchestra.run.smithi138.stdout:13 Jan 11:44:30 ntpd[12563]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-01-13T11:44:30.499 INFO:teuthology.orchestra.run.smithi138.stdout:13 Jan 11:44:30 ntpd[12563]: restrict ::: KOD does nothing without LIMITED. 2024-01-13T11:44:30.500 INFO:teuthology.orchestra.run.smithi138.stderr:restrict ::: KOD does nothing without LIMITED. 2024-01-13T11:44:30.501 INFO:teuthology.orchestra.run.smithi138.stdout:13 Jan 11:44:30 ntpd[12563]: Listen and drop on 0 v6wildcard [::]:123 2024-01-13T11:44:30.501 INFO:teuthology.orchestra.run.smithi138.stdout:13 Jan 11:44:30 ntpd[12563]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-01-13T11:44:30.501 INFO:teuthology.orchestra.run.smithi138.stdout:13 Jan 11:44:30 ntpd[12563]: Listen normally on 2 lo 127.0.0.1:123 2024-01-13T11:44:30.501 INFO:teuthology.orchestra.run.smithi138.stdout:13 Jan 11:44:30 ntpd[12563]: Listen normally on 3 enp3s0f1 172.21.15.138:123 2024-01-13T11:44:30.501 INFO:teuthology.orchestra.run.smithi138.stdout:13 Jan 11:44:30 ntpd[12563]: Listen normally on 4 lo [::1]:123 2024-01-13T11:44:30.501 INFO:teuthology.orchestra.run.smithi138.stdout:13 Jan 11:44:30 ntpd[12563]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6f71%5]:123 2024-01-13T11:44:30.501 INFO:teuthology.orchestra.run.smithi138.stdout:13 Jan 11:44:30 ntpd[12563]: Listening on routing socket on fd #22 for interface updates 2024-01-13T11:44:31.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:31 ntpd[12563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:31.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:31 ntpd[12563]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:31.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:31 ntpd[12563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:31.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:31 ntpd[12563]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:31.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:31 ntpd[12563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:31.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:31 ntpd[12563]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:31.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:31 ntpd[12563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-13T11:44:31.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:31 ntpd[12563]: can't open /var/log/ntpstats/peerstats.20240113: Permission denied 2024-01-13T11:44:32.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:32 ntpd[12577]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:32.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:32 ntpd[12577]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:32.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:32 ntpd[12577]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:32.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:32 ntpd[12577]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:32.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:32 ntpd[12577]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:32.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:32 ntpd[12577]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:32.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:32 ntpd[12577]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-13T11:44:32.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:32 ntpd[12577]: can't open /var/log/ntpstats/peerstats.20240113: Permission denied 2024-01-13T11:44:32.471 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:32 ntpd[12577]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:32.471 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:32 ntpd[12577]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:32.471 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:32 ntpd[12577]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-13T11:44:32.471 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:32 ntpd[12577]: can't open /var/log/ntpstats/peerstats.20240113: Permission denied 2024-01-13T11:44:33.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:33 ntpd[12577]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:33.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:33 ntpd[12577]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:33.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:33 ntpd[12577]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:33.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:33 ntpd[12577]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:33.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:33 ntpd[12577]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:33.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:33 ntpd[12577]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:33.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:33 ntpd[12577]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-13T11:44:33.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:33 ntpd[12577]: can't open /var/log/ntpstats/peerstats.20240113: Permission denied 2024-01-13T11:44:33.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:33 ntpd[12563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:33.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:33 ntpd[12563]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:33.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:33 ntpd[12563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:33.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:33 ntpd[12563]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:33.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:33 ntpd[12563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:33.500 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:33 ntpd[12563]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:33.500 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:33 ntpd[12563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-13T11:44:33.500 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:33 ntpd[12563]: can't open /var/log/ntpstats/peerstats.20240113: Permission denied 2024-01-13T11:44:33.500 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:33 ntpd[12563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:33.500 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:33 ntpd[12563]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:33.500 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:33 ntpd[12563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-13T11:44:33.500 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:33 ntpd[12563]: can't open /var/log/ntpstats/peerstats.20240113: Permission denied 2024-01-13T11:44:34.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:34 ntpd[12577]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:34.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:34 ntpd[12577]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:34.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:34 ntpd[12577]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:34.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:34 ntpd[12577]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:34.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:34 ntpd[12577]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:34.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:34 ntpd[12577]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:34.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:34 ntpd[12577]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-13T11:44:34.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:34 ntpd[12577]: can't open /var/log/ntpstats/peerstats.20240113: Permission denied 2024-01-13T11:44:34.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:34 ntpd[12577]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:34.471 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:34 ntpd[12577]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:34.471 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:34 ntpd[12577]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-13T11:44:34.471 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:34 ntpd[12577]: can't open /var/log/ntpstats/peerstats.20240113: Permission denied 2024-01-13T11:44:34.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:34 ntpd[12563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:34.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:34 ntpd[12563]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:34.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:34 ntpd[12563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:34.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:34 ntpd[12563]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:34.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:34 ntpd[12563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-13T11:44:34.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:34 ntpd[12563]: can't open /var/log/ntpstats/peerstats.20240113: Permission denied 2024-01-13T11:44:35.469 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:35 ntpd[12577]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:35.469 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:35 ntpd[12577]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:35.469 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:35 ntpd[12577]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:35.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:35 ntpd[12577]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:35.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:35 ntpd[12577]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-13T11:44:35.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:35 ntpd[12577]: can't open /var/log/ntpstats/peerstats.20240113: Permission denied 2024-01-13T11:44:35.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:35 ntpd[12563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:35.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:35 ntpd[12563]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:35.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:35 ntpd[12563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:35.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:35 ntpd[12563]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:35.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:35 ntpd[12563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:35.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:35 ntpd[12563]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:35.500 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:35 ntpd[12563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-13T11:44:35.500 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:35 ntpd[12563]: can't open /var/log/ntpstats/peerstats.20240113: Permission denied 2024-01-13T11:44:35.500 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:35 ntpd[12563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:35.500 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:35 ntpd[12563]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:35.500 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:35 ntpd[12563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-13T11:44:35.500 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:35 ntpd[12563]: can't open /var/log/ntpstats/peerstats.20240113: Permission denied 2024-01-13T11:44:36.469 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:36 ntpd[12577]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:36.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:36 ntpd[12577]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:36.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:36 ntpd[12577]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:36.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:36 ntpd[12577]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:36.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:36 ntpd[12577]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:36.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:36 ntpd[12577]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:36.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:36 ntpd[12577]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-13T11:44:36.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:36 ntpd[12577]: can't open /var/log/ntpstats/peerstats.20240113: Permission denied 2024-01-13T11:44:36.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:36 ntpd[12577]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:36.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:36 ntpd[12577]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:36.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:36 ntpd[12577]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-13T11:44:36.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:36 ntpd[12577]: can't open /var/log/ntpstats/peerstats.20240113: Permission denied 2024-01-13T11:44:36.498 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:36 ntpd[12563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:36.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:36 ntpd[12563]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:36.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:36 ntpd[12563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:36.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:36 ntpd[12563]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:36.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:36 ntpd[12563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-13T11:44:36.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:36 ntpd[12563]: can't open /var/log/ntpstats/peerstats.20240113: Permission denied 2024-01-13T11:44:37.469 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:37 ntpd[12577]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:37.469 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:37 ntpd[12577]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:37.469 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:37 ntpd[12577]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:37.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:37 ntpd[12577]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:37.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:37 ntpd[12577]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-13T11:44:37.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:37 ntpd[12577]: can't open /var/log/ntpstats/peerstats.20240113: Permission denied 2024-01-13T11:44:37.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:37 ntpd[12563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:37.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:37 ntpd[12563]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:37.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:37 ntpd[12563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:37.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:37 ntpd[12563]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:37.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:37 ntpd[12563]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:37.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:37 ntpd[12563]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:37.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:37 ntpd[12563]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-13T11:44:37.499 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:37 ntpd[12563]: can't open /var/log/ntpstats/peerstats.20240113: Permission denied 2024-01-13T11:44:37.500 INFO:teuthology.orchestra.run.smithi138.stdout:13 Jan 11:44:37 ntpd[12563]: ntpd: time slew +0.001523 s 2024-01-13T11:44:37.500 INFO:teuthology.orchestra.run.smithi138.stdout:ntpd: time slew +0.001523s 2024-01-13T11:44:37.500 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:37 ntpd[12563]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-01-13T11:44:37.500 INFO:teuthology.orchestra.run.smithi138.stderr:13 Jan 11:44:37 ntpd[12563]: can't open /var/log/ntpstats/loopstats.20240113: Permission denied 2024-01-13T11:44:37.557 INFO:teuthology.orchestra.run.smithi138.stdout: remote refid st t when poll reach delay offset jitter 2024-01-13T11:44:37.557 INFO:teuthology.orchestra.run.smithi138.stdout:============================================================================== 2024-01-13T11:44:37.557 INFO:teuthology.orchestra.run.smithi138.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-01-13T11:44:37.557 INFO:teuthology.orchestra.run.smithi138.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-01-13T11:44:37.557 INFO:teuthology.orchestra.run.smithi138.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-01-13T11:44:37.557 INFO:teuthology.orchestra.run.smithi138.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-01-13T11:44:38.469 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:38 ntpd[12577]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:38.469 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:38 ntpd[12577]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:38.469 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:38 ntpd[12577]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:38.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:38 ntpd[12577]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:38.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:38 ntpd[12577]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-13T11:44:38.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:38 ntpd[12577]: can't open /var/log/ntpstats/rawstats.20240113: Permission denied 2024-01-13T11:44:38.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:38 ntpd[12577]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-13T11:44:38.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:38 ntpd[12577]: can't open /var/log/ntpstats/peerstats.20240113: Permission denied 2024-01-13T11:44:38.470 INFO:teuthology.orchestra.run.smithi088.stdout:13 Jan 11:44:38 ntpd[12577]: ntpd: time slew -0.001373 s 2024-01-13T11:44:38.470 INFO:teuthology.orchestra.run.smithi088.stdout:ntpd: time slew -0.001373s 2024-01-13T11:44:38.470 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:38 ntpd[12577]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-01-13T11:44:38.471 INFO:teuthology.orchestra.run.smithi088.stderr:13 Jan 11:44:38 ntpd[12577]: can't open /var/log/ntpstats/loopstats.20240113: Permission denied 2024-01-13T11:44:38.531 INFO:teuthology.orchestra.run.smithi088.stdout: remote refid st t when poll reach delay offset jitter 2024-01-13T11:44:38.531 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================== 2024-01-13T11:44:38.531 INFO:teuthology.orchestra.run.smithi088.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-01-13T11:44:38.531 INFO:teuthology.orchestra.run.smithi088.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-01-13T11:44:38.531 INFO:teuthology.orchestra.run.smithi088.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-01-13T11:44:38.531 INFO:teuthology.orchestra.run.smithi088.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-01-13T11:44:38.532 INFO:teuthology.run_tasks:Running task nvme_loop... 2024-01-13T11:44:38.542 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2024-01-13T11:44:38.543 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-01-13T11:44:38.543 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/scratch_devs of=/dev/stdout 2024-01-13T11:44:38.582 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-13T11:44:38.582 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_1 2024-01-13T11:44:38.631 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-13T11:44:38.631 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-13T11:44:38.631 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 495 Links: 1 2024-01-13T11:44:38.631 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-13T11:44:38.631 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-01-13 11:43:01.149231396 +0000 2024-01-13T11:44:38.631 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-01-13 11:43:00.905237020 +0000 2024-01-13T11:44:38.631 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-01-13 11:43:00.905237020 +0000 2024-01-13T11:44:38.631 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: - 2024-01-13T11:44:38.632 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-13T11:44:38.686 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-01-13T11:44:38.686 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-01-13T11:44:38.686 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000438783 s, 1.2 MB/s 2024-01-13T11:44:38.687 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-13T11:44:38.741 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_2 2024-01-13T11:44:38.791 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-13T11:44:38.791 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-13T11:44:38.791 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 500 Links: 1 2024-01-13T11:44:38.791 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-13T11:44:38.791 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-01-13 11:43:01.605220886 +0000 2024-01-13T11:44:38.791 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-01-13 11:43:01.357226602 +0000 2024-01-13T11:44:38.791 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-01-13 11:43:01.357226602 +0000 2024-01-13T11:44:38.791 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: - 2024-01-13T11:44:38.792 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-13T11:44:38.846 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-01-13T11:44:38.846 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-01-13T11:44:38.846 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000438657 s, 1.2 MB/s 2024-01-13T11:44:38.847 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-13T11:44:38.896 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_3 2024-01-13T11:44:38.947 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-13T11:44:38.947 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-13T11:44:38.947 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 508 Links: 1 2024-01-13T11:44:38.947 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-13T11:44:38.947 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-01-13 11:43:01.825215814 +0000 2024-01-13T11:44:38.947 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-01-13 11:43:01.821215907 +0000 2024-01-13T11:44:38.947 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-01-13 11:43:01.821215907 +0000 2024-01-13T11:44:38.947 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: - 2024-01-13T11:44:38.947 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-13T11:44:39.002 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-01-13T11:44:39.003 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-01-13T11:44:39.003 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000396283 s, 1.3 MB/s 2024-01-13T11:44:39.003 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-13T11:44:39.056 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_4 2024-01-13T11:44:39.107 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-13T11:44:39.107 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-13T11:44:39.107 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 512 Links: 1 2024-01-13T11:44:39.107 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-13T11:44:39.107 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-01-13 11:43:02.521199772 +0000 2024-01-13T11:44:39.107 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-01-13 11:43:02.273205489 +0000 2024-01-13T11:44:39.107 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-01-13 11:43:02.273205489 +0000 2024-01-13T11:44:39.107 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: - 2024-01-13T11:44:39.108 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-13T11:44:39.162 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-01-13T11:44:39.163 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-01-13T11:44:39.163 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000372069 s, 1.4 MB/s 2024-01-13T11:44:39.163 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-13T11:44:39.216 DEBUG:teuthology.orchestra.run.smithi088:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-01-13T11:44:39.353 INFO:teuthology.orchestra.run.smithi088.stdout:loop 2024-01-13T11:44:39.355 INFO:tasks.nvme_loop:Connecting nvme_loop smithi088:/dev/vg_nvme/lv_1... 2024-01-13T11:44:39.355 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-01-13T11:44:39.379 INFO:teuthology.orchestra.run.smithi088.stdout:1 2024-01-13T11:44:39.406 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/vg_nvme/lv_11 2024-01-13T11:44:39.427 INFO:tasks.nvme_loop:Connecting nvme_loop smithi088:/dev/vg_nvme/lv_2... 2024-01-13T11:44:39.427 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-01-13T11:44:39.494 INFO:teuthology.orchestra.run.smithi088.stdout:1 2024-01-13T11:44:39.519 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/vg_nvme/lv_21 2024-01-13T11:44:39.543 INFO:tasks.nvme_loop:Connecting nvme_loop smithi088:/dev/vg_nvme/lv_3... 2024-01-13T11:44:39.544 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-01-13T11:44:39.614 INFO:teuthology.orchestra.run.smithi088.stdout:1 2024-01-13T11:44:39.638 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/vg_nvme/lv_31 2024-01-13T11:44:39.660 INFO:tasks.nvme_loop:Connecting nvme_loop smithi088:/dev/vg_nvme/lv_4... 2024-01-13T11:44:39.660 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-01-13T11:44:39.730 INFO:teuthology.orchestra.run.smithi088.stdout:1 2024-01-13T11:44:39.756 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/vg_nvme/lv_41 2024-01-13T11:44:39.779 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-01-13T11:44:39.780 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/scratch_devs of=/dev/stdout 2024-01-13T11:44:39.827 DEBUG:teuthology.orchestra.run.smithi088:> sudo nvme list 2024-01-13T11:44:39.882 INFO:teuthology.orchestra.run.smithi088.stdout:Node SN Model Namespace Usage Format FW Rev 2024-01-13T11:44:39.883 INFO:teuthology.orchestra.run.smithi088.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-01-13T11:44:39.883 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/nvme0n1 PHFT620400XD400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-01-13T11:44:39.883 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/nvme1n1 926d2812d55fa704acee Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-9 2024-01-13T11:44:39.883 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/nvme2n1 66ff27e154d3332366c5 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-9 2024-01-13T11:44:39.883 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/nvme3n1 38e73b7cdc6e832a4442 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-9 2024-01-13T11:44:39.883 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/nvme4n1 5bcbbb44462d7e8d0f9a Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-9 2024-01-13T11:44:39.884 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-01-13T11:44:39.884 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-01-13T11:44:39.884 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/scratch_devs 2024-01-13T11:44:39.942 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-01-13T11:44:39.942 DEBUG:teuthology.orchestra.run.smithi138:> dd if=/scratch_devs of=/dev/stdout 2024-01-13T11:44:39.951 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-13T11:44:39.951 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_1 2024-01-13T11:44:40.002 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-13T11:44:40.002 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-13T11:44:40.002 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 493 Links: 1 2024-01-13T11:44:40.002 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-13T11:44:40.003 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2024-01-13 11:43:01.720542968 +0000 2024-01-13T11:44:40.003 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2024-01-13 11:43:01.460548965 +0000 2024-01-13T11:44:40.003 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2024-01-13 11:43:01.460548965 +0000 2024-01-13T11:44:40.003 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: - 2024-01-13T11:44:40.003 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-13T11:44:40.058 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2024-01-13T11:44:40.058 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2024-01-13T11:44:40.058 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000430259 s, 1.2 MB/s 2024-01-13T11:44:40.059 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-13T11:44:40.112 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_2 2024-01-13T11:44:40.162 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-13T11:44:40.162 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-13T11:44:40.162 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 502 Links: 1 2024-01-13T11:44:40.162 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-13T11:44:40.162 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2024-01-13 11:43:01.940537895 +0000 2024-01-13T11:44:40.162 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2024-01-13 11:43:01.936537988 +0000 2024-01-13T11:44:40.162 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2024-01-13 11:43:01.936537988 +0000 2024-01-13T11:44:40.162 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: - 2024-01-13T11:44:40.163 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-13T11:44:40.218 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2024-01-13T11:44:40.218 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2024-01-13T11:44:40.218 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000394802 s, 1.3 MB/s 2024-01-13T11:44:40.219 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-13T11:44:40.272 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_3 2024-01-13T11:44:40.322 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-13T11:44:40.322 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-13T11:44:40.322 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 506 Links: 1 2024-01-13T11:44:40.322 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-13T11:44:40.322 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2024-01-13 11:43:02.636521844 +0000 2024-01-13T11:44:40.322 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2024-01-13 11:43:02.392527472 +0000 2024-01-13T11:44:40.322 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2024-01-13 11:43:02.392527472 +0000 2024-01-13T11:44:40.322 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: - 2024-01-13T11:44:40.323 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-13T11:44:40.377 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2024-01-13T11:44:40.377 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2024-01-13T11:44:40.377 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000444896 s, 1.2 MB/s 2024-01-13T11:44:40.378 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-13T11:44:40.428 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_4 2024-01-13T11:44:40.479 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-13T11:44:40.479 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-13T11:44:40.479 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 513 Links: 1 2024-01-13T11:44:40.479 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-13T11:44:40.479 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2024-01-13 11:43:03.100511145 +0000 2024-01-13T11:44:40.479 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2024-01-13 11:43:02.840517140 +0000 2024-01-13T11:44:40.479 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2024-01-13 11:43:02.840517140 +0000 2024-01-13T11:44:40.479 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: - 2024-01-13T11:44:40.480 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-13T11:44:40.537 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2024-01-13T11:44:40.537 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2024-01-13T11:44:40.538 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000378073 s, 1.4 MB/s 2024-01-13T11:44:40.538 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-13T11:44:40.591 DEBUG:teuthology.orchestra.run.smithi138:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-01-13T11:44:40.731 INFO:teuthology.orchestra.run.smithi138.stdout:loop 2024-01-13T11:44:40.733 INFO:tasks.nvme_loop:Connecting nvme_loop smithi138:/dev/vg_nvme/lv_1... 2024-01-13T11:44:40.733 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-01-13T11:44:40.753 INFO:teuthology.orchestra.run.smithi138.stdout:1 2024-01-13T11:44:40.779 INFO:teuthology.orchestra.run.smithi138.stdout:/dev/vg_nvme/lv_11 2024-01-13T11:44:40.803 INFO:tasks.nvme_loop:Connecting nvme_loop smithi138:/dev/vg_nvme/lv_2... 2024-01-13T11:44:40.803 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-01-13T11:44:40.873 INFO:teuthology.orchestra.run.smithi138.stdout:1 2024-01-13T11:44:40.897 INFO:teuthology.orchestra.run.smithi138.stdout:/dev/vg_nvme/lv_21 2024-01-13T11:44:40.917 INFO:tasks.nvme_loop:Connecting nvme_loop smithi138:/dev/vg_nvme/lv_3... 2024-01-13T11:44:40.917 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-01-13T11:44:40.981 INFO:teuthology.orchestra.run.smithi138.stdout:1 2024-01-13T11:44:41.007 INFO:teuthology.orchestra.run.smithi138.stdout:/dev/vg_nvme/lv_31 2024-01-13T11:44:41.030 INFO:tasks.nvme_loop:Connecting nvme_loop smithi138:/dev/vg_nvme/lv_4... 2024-01-13T11:44:41.030 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-01-13T11:44:41.092 INFO:teuthology.orchestra.run.smithi138.stdout:1 2024-01-13T11:44:41.120 INFO:teuthology.orchestra.run.smithi138.stdout:/dev/vg_nvme/lv_41 2024-01-13T11:44:41.143 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-01-13T11:44:41.144 DEBUG:teuthology.orchestra.run.smithi138:> dd if=/scratch_devs of=/dev/stdout 2024-01-13T11:44:41.193 DEBUG:teuthology.orchestra.run.smithi138:> sudo nvme list 2024-01-13T11:44:41.248 INFO:teuthology.orchestra.run.smithi138.stdout:Node SN Model Namespace Usage Format FW Rev 2024-01-13T11:44:41.248 INFO:teuthology.orchestra.run.smithi138.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-01-13T11:44:41.248 INFO:teuthology.orchestra.run.smithi138.stdout:/dev/nvme0n1 CVFT6161003C400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-01-13T11:44:41.248 INFO:teuthology.orchestra.run.smithi138.stdout:/dev/nvme1n1 44eedd88dd6e305b6719 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-9 2024-01-13T11:44:41.248 INFO:teuthology.orchestra.run.smithi138.stdout:/dev/nvme2n1 0bdfd76c51f876ffb3c3 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-9 2024-01-13T11:44:41.248 INFO:teuthology.orchestra.run.smithi138.stdout:/dev/nvme3n1 702202b53bf9234cc21e Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-9 2024-01-13T11:44:41.248 INFO:teuthology.orchestra.run.smithi138.stdout:/dev/nvme4n1 72a3971563a9aab3c1b3 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-9 2024-01-13T11:44:41.249 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-01-13T11:44:41.249 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-01-13T11:44:41.249 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/scratch_devs 2024-01-13T11:44:41.306 INFO:teuthology.run_tasks:Running task cephadm... 2024-01-13T11:44:41.389 INFO:tasks.cephadm:Config: {'roleless': True, 'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '445562ab4bc3ddfb386936119050695810860bcb'} 2024-01-13T11:44:41.390 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb 2024-01-13T11:44:41.390 INFO:tasks.cephadm:Cluster fsid is 236db248-b209-11ee-95ac-87774f69a715 2024-01-13T11:44:41.390 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-01-13T11:44:41.390 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-01-13T11:44:41.390 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi088': '172.21.15.88', 'mon.smithi138': '172.21.15.138'} 2024-01-13T11:44:41.391 INFO:tasks.cephadm:Normalizing hostnames... 2024-01-13T11:44:41.391 DEBUG:teuthology.orchestra.run.smithi088:> sudo hostname $(hostname -s) 2024-01-13T11:44:41.406 DEBUG:teuthology.orchestra.run.smithi138:> sudo hostname $(hostname -s) 2024-01-13T11:44:41.423 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph-ci.git ref 445562ab4bc3ddfb386936119050695810860bcb)... 2024-01-13T11:44:41.423 DEBUG:teuthology.orchestra.run.smithi088:> curl --silent https://raw.githubusercontent.com/ceph/ceph-ci/445562ab4bc3ddfb386936119050695810860bcb/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-13T11:44:41.678 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-rw-r-- 1 ubuntu ubuntu 355729 Jan 13 11:44 /home/ubuntu/cephtest/cephadm 2024-01-13T11:44:41.678 DEBUG:teuthology.orchestra.run.smithi138:> curl --silent https://raw.githubusercontent.com/ceph/ceph-ci/445562ab4bc3ddfb386936119050695810860bcb/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-13T11:44:41.829 INFO:teuthology.orchestra.run.smithi138.stdout:-rw-rw-r-- 1 ubuntu ubuntu 355729 Jan 13 11:44 /home/ubuntu/cephtest/cephadm 2024-01-13T11:44:41.829 DEBUG:teuthology.orchestra.run.smithi088:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-13T11:44:41.838 DEBUG:teuthology.orchestra.run.smithi138:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-13T11:44:41.853 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /etc/ceph 2024-01-13T11:44:41.895 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /etc/ceph 2024-01-13T11:44:41.911 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 777 /etc/ceph 2024-01-13T11:44:41.952 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 777 /etc/ceph 2024-01-13T11:44:41.969 INFO:tasks.cephadm:Writing seed config... 2024-01-13T11:44:41.970 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-01-13T11:44:41.970 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-01-13T11:44:41.970 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-01-13T11:44:41.970 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-01-13T11:44:41.970 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-01-13T11:44:41.971 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-01-13T11:44:41.971 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-01-13T11:44:41.971 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-01-13T11:44:41.971 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-01-13T11:44:41.971 DEBUG:teuthology.orchestra.run.smithi088:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-01-13T11:44:41.999 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 236db248-b209-11ee-95ac-87774f69a715 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-01-13T11:44:41.999 DEBUG:teuthology.orchestra.run.smithi088:mon.smithi088> sudo journalctl -f -n 0 -u ceph-236db248-b209-11ee-95ac-87774f69a715@mon.smithi088.service 2024-01-13T11:44:42.042 INFO:tasks.cephadm:Bootstrapping... 2024-01-13T11:44:42.043 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb -v bootstrap --fsid 236db248-b209-11ee-95ac-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.88 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-01-13T11:44:42.288 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2024-01-13T11:44:42.418 INFO:teuthology.orchestra.run.smithi088.stderr:-------------------------------------------------------------------------------- 2024-01-13T11:44:42.419 INFO:teuthology.orchestra.run.smithi088.stderr:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb', '-v', 'bootstrap', '--fsid', '236db248-b209-11ee-95ac-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.88', '--skip-admin-label'] 2024-01-13T11:44:42.419 INFO:teuthology.orchestra.run.smithi088.stderr:Verifying podman|docker is present... 2024-01-13T11:44:42.419 INFO:teuthology.orchestra.run.smithi088.stderr:Verifying lvm2 is present... 2024-01-13T11:44:42.420 INFO:teuthology.orchestra.run.smithi088.stderr:Verifying time synchronization is in place... 2024-01-13T11:44:42.429 INFO:teuthology.orchestra.run.smithi088.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-01-13T11:44:42.436 INFO:teuthology.orchestra.run.smithi088.stderr:systemctl: inactive 2024-01-13T11:44:42.443 INFO:teuthology.orchestra.run.smithi088.stderr:systemctl: Failed to get unit file state for chronyd.service: No such file or directory 2024-01-13T11:44:42.450 INFO:teuthology.orchestra.run.smithi088.stderr:systemctl: inactive 2024-01-13T11:44:42.456 INFO:teuthology.orchestra.run.smithi088.stderr:systemctl: masked 2024-01-13T11:44:42.462 INFO:teuthology.orchestra.run.smithi088.stderr:systemctl: inactive 2024-01-13T11:44:42.469 INFO:teuthology.orchestra.run.smithi088.stderr:systemctl: Failed to get unit file state for ntpd.service: No such file or directory 2024-01-13T11:44:42.475 INFO:teuthology.orchestra.run.smithi088.stderr:systemctl: inactive 2024-01-13T11:44:42.482 INFO:teuthology.orchestra.run.smithi088.stderr:systemctl: enabled 2024-01-13T11:44:42.486 INFO:teuthology.orchestra.run.smithi088.stderr:systemctl: active 2024-01-13T11:44:42.487 INFO:teuthology.orchestra.run.smithi088.stderr:Unit ntp.service is enabled and running 2024-01-13T11:44:42.487 INFO:teuthology.orchestra.run.smithi088.stderr:Repeating the final host check... 2024-01-13T11:44:42.488 INFO:teuthology.orchestra.run.smithi088.stderr:docker (/usr/bin/docker) is present 2024-01-13T11:44:42.488 INFO:teuthology.orchestra.run.smithi088.stderr:systemctl is present 2024-01-13T11:44:42.488 INFO:teuthology.orchestra.run.smithi088.stderr:lvcreate is present 2024-01-13T11:44:42.493 INFO:teuthology.orchestra.run.smithi088.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-01-13T11:44:42.499 INFO:teuthology.orchestra.run.smithi088.stderr:systemctl: inactive 2024-01-13T11:44:42.506 INFO:teuthology.orchestra.run.smithi088.stderr:systemctl: Failed to get unit file state for chronyd.service: No such file or directory 2024-01-13T11:44:42.511 INFO:teuthology.orchestra.run.smithi088.stderr:systemctl: inactive 2024-01-13T11:44:42.517 INFO:teuthology.orchestra.run.smithi088.stderr:systemctl: masked 2024-01-13T11:44:42.524 INFO:teuthology.orchestra.run.smithi088.stderr:systemctl: inactive 2024-01-13T11:44:42.531 INFO:teuthology.orchestra.run.smithi088.stderr:systemctl: Failed to get unit file state for ntpd.service: No such file or directory 2024-01-13T11:44:42.537 INFO:teuthology.orchestra.run.smithi088.stderr:systemctl: inactive 2024-01-13T11:44:42.545 INFO:teuthology.orchestra.run.smithi088.stderr:systemctl: enabled 2024-01-13T11:44:42.552 INFO:teuthology.orchestra.run.smithi088.stderr:systemctl: active 2024-01-13T11:44:42.554 INFO:teuthology.orchestra.run.smithi088.stderr:Unit ntp.service is enabled and running 2024-01-13T11:44:42.554 INFO:teuthology.orchestra.run.smithi088.stderr:Host looks OK 2024-01-13T11:44:42.554 INFO:teuthology.orchestra.run.smithi088.stderr:Cluster fsid: 236db248-b209-11ee-95ac-87774f69a715 2024-01-13T11:44:42.554 INFO:teuthology.orchestra.run.smithi088.stderr:Acquiring lock 140644167924128 on /run/cephadm/236db248-b209-11ee-95ac-87774f69a715.lock 2024-01-13T11:44:42.554 INFO:teuthology.orchestra.run.smithi088.stderr:Lock 140644167924128 acquired on /run/cephadm/236db248-b209-11ee-95ac-87774f69a715.lock 2024-01-13T11:44:42.555 INFO:teuthology.orchestra.run.smithi088.stderr:Verifying IP 172.21.15.88 port 3300 ... 2024-01-13T11:44:42.555 INFO:teuthology.orchestra.run.smithi088.stderr:Verifying IP 172.21.15.88 port 6789 ... 2024-01-13T11:44:42.555 INFO:teuthology.orchestra.run.smithi088.stderr:Base mon IP(s) is [172.21.15.88:3300, 172.21.15.88:6789], mon addrv is [v2:172.21.15.88:3300,v1:172.21.15.88:6789] 2024-01-13T11:44:42.560 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/sbin/ip: default via 172.21.15.254 dev enp3s0f1 2024-01-13T11:44:42.560 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.88 metric 100 2024-01-13T11:44:42.561 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/sbin/ip: 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 2024-01-13T11:44:42.561 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.88 2024-01-13T11:44:42.561 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/sbin/ip: 172.21.15.254 dev enp3s0f1 proto dhcp scope link src 172.21.15.88 metric 100 2024-01-13T11:44:42.565 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-01-13T11:44:42.566 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-01-13T11:44:42.566 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 expires 1785sec pref medium 2024-01-13T11:44:42.570 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-01-13T11:44:42.570 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/sbin/ip: inet6 ::1/128 scope host 2024-01-13T11:44:42.570 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/sbin/ip: valid_lft forever preferred_lft forever 2024-01-13T11:44:42.570 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-01-13T11:44:42.570 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/sbin/ip: inet6 fe80::ec4:7aff:fe8f:d42d/64 scope link 2024-01-13T11:44:42.571 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/sbin/ip: valid_lft forever preferred_lft forever 2024-01-13T11:44:42.572 INFO:teuthology.orchestra.run.smithi088.stderr:Mon IP `172.21.15.88` is in CIDR network `172.21.0.0/20` 2024-01-13T11:44:42.572 INFO:teuthology.orchestra.run.smithi088.stderr:Mon IP `172.21.15.88` is in CIDR network `172.21.0.0/20` 2024-01-13T11:44:42.572 INFO:teuthology.orchestra.run.smithi088.stderr:Mon IP `172.21.15.88` is in CIDR network `172.21.15.254/32` 2024-01-13T11:44:42.572 INFO:teuthology.orchestra.run.smithi088.stderr:Mon IP `172.21.15.88` is in CIDR network `172.21.15.254/32` 2024-01-13T11:44:42.572 INFO:teuthology.orchestra.run.smithi088.stderr:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20', '172.21.15.254/32', '172.21.15.254/32'] 2024-01-13T11:44:42.573 INFO:teuthology.orchestra.run.smithi088.stderr:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-01-13T11:44:42.573 INFO:teuthology.orchestra.run.smithi088.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb... 2024-01-13T11:44:42.977 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/docker: 445562ab4bc3ddfb386936119050695810860bcb: Pulling from ceph-ci/ceph 2024-01-13T11:44:42.986 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/docker: 6c507150f7e9: Pulling fs layer 2024-01-13T11:44:42.987 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/docker: 9bc82cfe8091: Pulling fs layer 2024-01-13T11:44:49.972 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/docker: 6c507150f7e9: Download complete 2024-01-13T11:44:53.777 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/docker: 6c507150f7e9: Pull complete 2024-01-13T11:45:01.854 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/docker: 9bc82cfe8091: Verifying Checksum 2024-01-13T11:45:01.855 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/docker: 9bc82cfe8091: Download complete 2024-01-13T11:45:13.862 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/docker: 9bc82cfe8091: Pull complete 2024-01-13T11:45:14.011 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/docker: Digest: sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 2024-01-13T11:45:14.063 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/docker: Status: Downloaded newer image for quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb 2024-01-13T11:45:14.102 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/docker: quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb 2024-01-13T11:45:24.811 INFO:teuthology.orchestra.run.smithi088.stderr:ceph: ceph version 16.2.14-456-g445562ab (445562ab4bc3ddfb386936119050695810860bcb) pacific (stable) 2024-01-13T11:45:25.435 INFO:teuthology.orchestra.run.smithi088.stderr:Ceph version: ceph version 16.2.14-456-g445562ab (445562ab4bc3ddfb386936119050695810860bcb) pacific (stable) 2024-01-13T11:45:25.435 INFO:teuthology.orchestra.run.smithi088.stderr:Extracting ceph user uid/gid from container image... 2024-01-13T11:45:26.207 INFO:teuthology.orchestra.run.smithi088.stderr:stat: 167 167 2024-01-13T11:45:26.788 INFO:teuthology.orchestra.run.smithi088.stderr:Creating initial keys... 2024-01-13T11:45:27.529 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-authtool: AQDXd6Jl5QfyHxAAn4X/4lubGh7v7wrQlERoSg== 2024-01-13T11:45:28.987 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-authtool: AQDYd6JlDzNBOxAA5dJa1g86JHO4l1X3Nbm7IQ== 2024-01-13T11:45:30.450 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-authtool: AQDad6Jl8MFGGxAAwTA9NCXXB+MVHqrqkzTBnw== 2024-01-13T11:45:31.036 INFO:teuthology.orchestra.run.smithi088.stderr:Creating initial monmap... 2024-01-13T11:45:31.824 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-13T11:45:31.824 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 236db248-b209-11ee-95ac-87774f69a715 2024-01-13T11:45:31.824 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-13T11:45:32.381 INFO:teuthology.orchestra.run.smithi088.stderr:monmaptool for smithi088 [v2:172.21.15.88:3300,v1:172.21.15.88:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-13T11:45:32.382 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/monmaptool: set fsid to 236db248-b209-11ee-95ac-87774f69a715 2024-01-13T11:45:32.382 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-13T11:45:32.382 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-01-13T11:45:32.382 INFO:teuthology.orchestra.run.smithi088.stderr:Creating mon... 2024-01-13T11:45:33.161 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.164+0000 7f0947054880 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-13T11:45:33.162 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.164+0000 7f0947054880 1 imported monmap: 2024-01-13T11:45:33.162 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: epoch 0 2024-01-13T11:45:33.163 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: fsid 236db248-b209-11ee-95ac-87774f69a715 2024-01-13T11:45:33.163 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: last_changed 2024-01-13T11:45:31.831492+0000 2024-01-13T11:45:33.163 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: created 2024-01-13T11:45:31.831492+0000 2024-01-13T11:45:33.163 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-01-13T11:45:33.163 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-01-13T11:45:33.163 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.smithi088 2024-01-13T11:45:33.163 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: 2024-01-13T11:45:33.166 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.164+0000 7f0947054880 0 /usr/bin/ceph-mon: set fsid to 236db248-b209-11ee-95ac-87774f69a715 2024-01-13T11:45:33.166 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: RocksDB version: 6.8.1 2024-01-13T11:45:33.166 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: 2024-01-13T11:45:33.166 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-13T11:45:33.166 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Compile date Jan 12 2024 2024-01-13T11:45:33.167 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: DB SUMMARY 2024-01-13T11:45:33.167 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: 2024-01-13T11:45:33.167 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi088/store.db dir, Total Num: 0, files: 2024-01-13T11:45:33.167 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: 2024-01-13T11:45:33.167 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi088/store.db: 2024-01-13T11:45:33.167 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: 2024-01-13T11:45:33.167 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.error_if_exists: 0 2024-01-13T11:45:33.167 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.create_if_missing: 1 2024-01-13T11:45:33.167 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.paranoid_checks: 1 2024-01-13T11:45:33.167 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.env: 0x55c65adc3080 2024-01-13T11:45:33.167 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.fs: Posix File System 2024-01-13T11:45:33.167 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.info_log: 0x55c65c7cb380 2024-01-13T11:45:33.167 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-13T11:45:33.167 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.statistics: (nil) 2024-01-13T11:45:33.168 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.use_fsync: 0 2024-01-13T11:45:33.168 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.max_log_file_size: 0 2024-01-13T11:45:33.168 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-13T11:45:33.191 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-13T11:45:33.191 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-13T11:45:33.192 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-13T11:45:33.192 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.allow_fallocate: 1 2024-01-13T11:45:33.192 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-13T11:45:33.192 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-13T11:45:33.192 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.use_direct_reads: 0 2024-01-13T11:45:33.192 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-13T11:45:33.192 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.create_missing_column_families: 0 2024-01-13T11:45:33.192 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.db_log_dir: 2024-01-13T11:45:33.195 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi088/store.db 2024-01-13T11:45:33.195 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-13T11:45:33.195 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.max_subcompactions: 1 2024-01-13T11:45:33.195 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.max_background_flushes: -1 2024-01-13T11:45:33.195 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-13T11:45:33.195 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-13T11:45:33.195 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-13T11:45:33.195 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-13T11:45:33.196 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-13T11:45:33.196 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.advise_random_on_open: 1 2024-01-13T11:45:33.196 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-13T11:45:33.196 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.write_buffer_manager: 0x55c65c7d1980 2024-01-13T11:45:33.196 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-13T11:45:33.196 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-13T11:45:33.196 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-13T11:45:33.196 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-13T11:45:33.196 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.rate_limiter: (nil) 2024-01-13T11:45:33.196 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-13T11:45:33.196 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-13T11:45:33.196 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-13T11:45:33.196 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-13T11:45:33.197 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.unordered_write: 0 2024-01-13T11:45:33.197 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-13T11:45:33.197 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-13T11:45:33.197 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-13T11:45:33.197 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-13T11:45:33.197 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.row_cache: None 2024-01-13T11:45:33.197 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.wal_filter: None 2024-01-13T11:45:33.197 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-13T11:45:33.197 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-13T11:45:33.197 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.preserve_deletes: 0 2024-01-13T11:45:33.197 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.two_write_queues: 0 2024-01-13T11:45:33.197 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.manual_wal_flush: 0 2024-01-13T11:45:33.197 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.atomic_flush: 0 2024-01-13T11:45:33.197 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-13T11:45:33.197 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-13T11:45:33.198 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-13T11:45:33.198 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.log_readahead_size: 0 2024-01-13T11:45:33.198 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-13T11:45:33.198 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.max_background_jobs: 2 2024-01-13T11:45:33.198 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.max_background_compactions: -1 2024-01-13T11:45:33.198 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-13T11:45:33.198 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-13T11:45:33.198 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-13T11:45:33.198 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.max_total_wal_size: 0 2024-01-13T11:45:33.198 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-13T11:45:33.198 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-13T11:45:33.198 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-13T11:45:33.199 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-13T11:45:33.199 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.max_open_files: -1 2024-01-13T11:45:33.199 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.bytes_per_sync: 0 2024-01-13T11:45:33.199 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-13T11:45:33.199 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-13T11:45:33.199 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-13T11:45:33.199 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Compression algorithms supported: 2024-01-13T11:45:33.199 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-13T11:45:33.200 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: kZSTD supported: 0 2024-01-13T11:45:33.200 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: kXpressCompression supported: 0 2024-01-13T11:45:33.200 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-13T11:45:33.200 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: kLZ4Compression supported: 1 2024-01-13T11:45:33.200 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: kBZip2Compression supported: 0 2024-01-13T11:45:33.200 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: kZlibCompression supported: 1 2024-01-13T11:45:33.200 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: kSnappyCompression supported: 1 2024-01-13T11:45:33.200 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-13T11:45:33.200 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-01-13T11:45:33.200 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: 2024-01-13T11:45:33.200 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi088/store.db/MANIFEST-000001 2024-01-13T11:45:33.200 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: 2024-01-13T11:45:33.200 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-13T11:45:33.201 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: 2024-01-13T11:45:33.201 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-13T11:45:33.201 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.merge_operator: 2024-01-13T11:45:33.201 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.compaction_filter: None 2024-01-13T11:45:33.201 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.compaction_filter_factory: None 2024-01-13T11:45:33.201 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-13T11:45:33.201 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-13T11:45:33.201 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c65c6dfd30) 2024-01-13T11:45:33.201 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-01-13T11:45:33.201 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-13T11:45:33.201 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-13T11:45:33.201 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-01-13T11:45:33.201 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: index_type: 0 2024-01-13T11:45:33.201 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-01-13T11:45:33.201 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-01-13T11:45:33.202 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-01-13T11:45:33.202 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-01-13T11:45:33.202 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: checksum: 1 2024-01-13T11:45:33.202 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-01-13T11:45:33.202 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: block_cache: 0x55c65c716d10 2024-01-13T11:45:33.202 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-01-13T11:45:33.202 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-01-13T11:45:33.202 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-01-13T11:45:33.202 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-01-13T11:45:33.202 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-01-13T11:45:33.202 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-01-13T11:45:33.202 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-01-13T11:45:33.202 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-01-13T11:45:33.203 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-01-13T11:45:33.203 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-01-13T11:45:33.203 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-01-13T11:45:33.203 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-01-13T11:45:33.203 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-01-13T11:45:33.203 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-01-13T11:45:33.203 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-01-13T11:45:33.204 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-13T11:45:33.204 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-01-13T11:45:33.204 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-01-13T11:45:33.204 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-01-13T11:45:33.204 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: format_version: 2 2024-01-13T11:45:33.204 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-01-13T11:45:33.204 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: block_align: 0 2024-01-13T11:45:33.204 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: 2024-01-13T11:45:33.204 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-13T11:45:33.204 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-13T11:45:33.204 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.compression: NoCompression 2024-01-13T11:45:33.204 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-13T11:45:33.204 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-13T11:45:33.204 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-13T11:45:33.205 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.num_levels: 7 2024-01-13T11:45:33.205 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-13T11:45:33.205 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-13T11:45:33.205 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-13T11:45:33.205 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-13T11:45:33.205 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-13T11:45:33.205 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-13T11:45:33.205 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-13T11:45:33.205 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-13T11:45:33.205 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-13T11:45:33.205 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-13T11:45:33.205 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.compression_opts.level: 32767 2024-01-13T11:45:33.205 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-13T11:45:33.205 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-13T11:45:33.206 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-13T11:45:33.206 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.compression_opts.enabled: false 2024-01-13T11:45:33.206 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-13T11:45:33.206 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-13T11:45:33.206 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-13T11:45:33.206 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-13T11:45:33.206 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-13T11:45:33.206 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-13T11:45:33.206 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-13T11:45:33.206 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-13T11:45:33.206 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-13T11:45:33.206 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-13T11:45:33.207 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-13T11:45:33.207 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-13T11:45:33.207 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-13T11:45:33.207 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-13T11:45:33.207 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-13T11:45:33.207 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-13T11:45:33.207 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-13T11:45:33.207 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.arena_block_size: 4194304 2024-01-13T11:45:33.207 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-13T11:45:33.208 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-13T11:45:33.208 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-13T11:45:33.208 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-13T11:45:33.208 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-13T11:45:33.208 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-13T11:45:33.208 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-13T11:45:33.208 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-13T11:45:33.208 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-13T11:45:33.208 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-13T11:45:33.208 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-13T11:45:33.208 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-13T11:45:33.208 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-13T11:45:33.208 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-13T11:45:33.208 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.table_properties_collectors: 2024-01-13T11:45:33.209 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.inplace_update_support: 0 2024-01-13T11:45:33.209 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-13T11:45:33.209 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-13T11:45:33.209 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-13T11:45:33.209 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-13T11:45:33.209 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.bloom_locality: 0 2024-01-13T11:45:33.209 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.max_successive_merges: 0 2024-01-13T11:45:33.209 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-13T11:45:33.209 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-13T11:45:33.209 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.force_consistency_checks: 0 2024-01-13T11:45:33.209 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-13T11:45:33.209 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.ttl: 2592000 2024-01-13T11:45:33.209 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-13T11:45:33.209 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi088/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-13T11:45:33.210 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: 2024-01-13T11:45:33.210 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.168+0000 7f0947054880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-01-13T11:45:33.210 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: 2024-01-13T11:45:33.210 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.172+0000 7f0947054880 4 rocksdb: DB pointer 0x55c65c7df800 2024-01-13T11:45:33.210 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.172+0000 7f092fec9700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-13T11:45:33.210 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.172+0000 7f092fec9700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-13T11:45:33.210 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-01-13T11:45:33.210 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T11:45:33.210 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-13T11:45:33.210 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-13T11:45:33.210 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-13T11:45:33.210 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-13T11:45:33.210 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-13T11:45:33.211 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-13T11:45:33.211 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: 2024-01-13T11:45:33.211 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-13T11:45:33.211 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T11:45:33.211 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T11:45:33.211 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-13T11:45:33.211 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-13T11:45:33.211 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: 2024-01-13T11:45:33.212 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-13T11:45:33.212 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T11:45:33.212 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T11:45:33.212 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T11:45:33.212 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-01-13T11:45:33.212 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-13T11:45:33.212 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-01-13T11:45:33.212 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-01-13T11:45:33.212 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-01-13T11:45:33.212 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T11:45:33.212 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T11:45:33.212 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-13T11:45:33.212 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: 2024-01-13T11:45:33.213 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-01-13T11:45:33.213 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: 2024-01-13T11:45:33.213 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-13T11:45:33.213 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T11:45:33.213 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T11:45:33.213 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-13T11:45:33.213 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-13T11:45:33.213 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: 2024-01-13T11:45:33.213 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-13T11:45:33.213 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T11:45:33.213 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T11:45:33.213 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T11:45:33.213 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-01-13T11:45:33.213 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-13T11:45:33.214 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-01-13T11:45:33.214 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-01-13T11:45:33.214 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-01-13T11:45:33.214 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T11:45:33.214 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T11:45:33.214 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-13T11:45:33.214 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: 2024-01-13T11:45:33.214 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-01-13T11:45:33.214 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: 2024-01-13T11:45:33.214 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.176+0000 7f0947054880 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-01-13T11:45:33.214 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.176+0000 7f0947054880 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-01-13T11:45:33.215 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph-mon: debug 2024-01-13T11:45:33.176+0000 7f0947054880 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi088 for mon.smithi088 2024-01-13T11:45:33.744 INFO:teuthology.orchestra.run.smithi088.stderr:create mon.smithi088 on 2024-01-13T11:45:33.978 INFO:teuthology.orchestra.run.smithi088.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /etc/systemd/system/ceph.target. 2024-01-13T11:45:34.186 INFO:teuthology.orchestra.run.smithi088.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-236db248-b209-11ee-95ac-87774f69a715.target -> /etc/systemd/system/ceph-236db248-b209-11ee-95ac-87774f69a715.target. 2024-01-13T11:45:34.186 INFO:teuthology.orchestra.run.smithi088.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-236db248-b209-11ee-95ac-87774f69a715.target -> /etc/systemd/system/ceph-236db248-b209-11ee-95ac-87774f69a715.target. 2024-01-13T11:45:34.614 INFO:teuthology.orchestra.run.smithi088.stderr:systemctl: Failed to reset failed state of unit ceph-236db248-b209-11ee-95ac-87774f69a715@mon.smithi088.service: Unit ceph-236db248-b209-11ee-95ac-87774f69a715@mon.smithi088.service not loaded. 2024-01-13T11:45:34.622 INFO:teuthology.orchestra.run.smithi088.stderr:systemctl: Created symlink /etc/systemd/system/ceph-236db248-b209-11ee-95ac-87774f69a715.target.wants/ceph-236db248-b209-11ee-95ac-87774f69a715@mon.smithi088.service -> /etc/systemd/system/ceph-236db248-b209-11ee-95ac-87774f69a715@.service. 2024-01-13T11:45:34.829 INFO:teuthology.orchestra.run.smithi088.stderr:firewalld does not appear to be present 2024-01-13T11:45:34.829 INFO:teuthology.orchestra.run.smithi088.stderr:Not possible to enable service . firewalld.service is not available 2024-01-13T11:45:34.829 INFO:teuthology.orchestra.run.smithi088.stderr:Waiting for mon to start... 2024-01-13T11:45:34.829 INFO:teuthology.orchestra.run.smithi088.stderr:Waiting for mon... 2024-01-13T11:45:36.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:36 smithi088 bash[13541]: cluster 2024-01-13T11:45:36.207423+0000 mon.smithi088 (mon.0) 1 : cluster [INF] mon.smithi088 is new leader, mons smithi088 in quorum (ranks 0) 2024-01-13T11:45:36.592 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: cluster: 2024-01-13T11:45:36.592 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: id: 236db248-b209-11ee-95ac-87774f69a715 2024-01-13T11:45:36.592 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-01-13T11:45:36.592 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: 2024-01-13T11:45:36.592 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: services: 2024-01-13T11:45:36.592 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi088 (age 0.364231s) 2024-01-13T11:45:36.592 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: mgr: no daemons active 2024-01-13T11:45:36.592 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-01-13T11:45:36.592 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: 2024-01-13T11:45:36.592 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: data: 2024-01-13T11:45:36.593 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-01-13T11:45:36.593 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-01-13T11:45:36.593 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-01-13T11:45:36.593 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: pgs: 2024-01-13T11:45:36.593 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: 2024-01-13T11:45:37.178 INFO:teuthology.orchestra.run.smithi088.stderr:mon is available 2024-01-13T11:45:37.178 INFO:teuthology.orchestra.run.smithi088.stderr:Assimilating anything we can from ceph.conf... 2024-01-13T11:45:37.645 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:37 smithi088 bash[13541]: cluster 2024-01-13T11:45:36.233661+0000 mon.smithi088 (mon.0) 2 : cluster [INF] mon.smithi088 is new leader, mons smithi088 in quorum (ranks 0) 2024-01-13T11:45:37.645 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:37 smithi088 bash[13541]: cluster 2024-01-13T11:45:36.233993+0000 mon.smithi088 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi088=[v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0]} removed_ranks: {} 2024-01-13T11:45:37.645 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:37 smithi088 bash[13541]: cluster 2024-01-13T11:45:36.239141+0000 mon.smithi088 (mon.0) 4 : cluster [DBG] fsmap 2024-01-13T11:45:37.645 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:37 smithi088 bash[13541]: cluster 2024-01-13T11:45:36.243516+0000 mon.smithi088 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-01-13T11:45:37.645 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:37 smithi088 bash[13541]: cluster 2024-01-13T11:45:36.244334+0000 mon.smithi088 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-01-13T11:45:37.645 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:37 smithi088 bash[13541]: audit 2024-01-13T11:45:36.597850+0000 mon.smithi088 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.88:0/4021169819' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-01-13T11:45:38.348 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: 2024-01-13T11:45:38.348 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: [global] 2024-01-13T11:45:38.348 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: fsid = 236db248-b209-11ee-95ac-87774f69a715 2024-01-13T11:45:38.348 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.88:3300,v1:172.21.15.88:6789] 2024-01-13T11:45:38.348 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-01-13T11:45:38.348 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-01-13T11:45:38.348 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-01-13T11:45:38.348 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-01-13T11:45:38.349 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: 2024-01-13T11:45:38.349 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: [mgr] 2024-01-13T11:45:38.349 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-01-13T11:45:38.349 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: 2024-01-13T11:45:38.349 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: [osd] 2024-01-13T11:45:38.349 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-01-13T11:45:38.349 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-01-13T11:45:38.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:38 smithi088 bash[13541]: audit 2024-01-13T11:45:38.349054+0000 mon.smithi088 (mon.0) 8 : audit [INF] from='client.? 172.21.15.88:0/3999649749' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-13T11:45:38.645 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:38 smithi088 bash[13541]: audit 2024-01-13T11:45:38.353580+0000 mon.smithi088 (mon.0) 9 : audit [INF] from='client.? 172.21.15.88:0/3999649749' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-01-13T11:45:38.947 INFO:teuthology.orchestra.run.smithi088.stderr:Generating new minimal ceph.conf... 2024-01-13T11:45:40.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:40 smithi088 bash[13541]: audit 2024-01-13T11:45:39.969382+0000 mon.smithi088 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.88:0/1880630501' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:45:40.563 INFO:teuthology.orchestra.run.smithi088.stderr:Restarting the monitor... 2024-01-13T11:45:40.895 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:40 smithi088 systemd[1]: Stopping Ceph mon.smithi088 for 236db248-b209-11ee-95ac-87774f69a715... 2024-01-13T11:45:40.895 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:40 smithi088 bash[13932]: Error response from daemon: No such container: ceph-236db248-b209-11ee-95ac-87774f69a715-mon.smithi088 2024-01-13T11:45:40.895 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:40 smithi088 bash[13541]: debug 2024-01-13T11:45:40.752+0000 7f80d0104700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.smithi088 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-13T11:45:40.895 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:40 smithi088 bash[13541]: debug 2024-01-13T11:45:40.752+0000 7f80d0104700 -1 mon.smithi088@0(leader) e1 *** Got Signal Terminated *** 2024-01-13T11:45:41.340 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:41 smithi088 bash[13968]: ceph-236db248-b209-11ee-95ac-87774f69a715-mon-smithi088 2024-01-13T11:45:41.523 INFO:teuthology.orchestra.run.smithi088.stderr:Setting public_network to 172.21.15.254/32,172.21.0.0/20 in mon config section 2024-01-13T11:45:41.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:41 smithi088 bash[14030]: Error response from daemon: No such container: ceph-236db248-b209-11ee-95ac-87774f69a715-mon-smithi088 2024-01-13T11:45:41.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:41 smithi088 systemd[1]: ceph-236db248-b209-11ee-95ac-87774f69a715@mon.smithi088.service: Succeeded. 2024-01-13T11:45:41.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:41 smithi088 systemd[1]: Stopped Ceph mon.smithi088 for 236db248-b209-11ee-95ac-87774f69a715. 2024-01-13T11:45:41.645 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:41 smithi088 systemd[1]: Started Ceph mon.smithi088 for 236db248-b209-11ee-95ac-87774f69a715. 2024-01-13T11:45:42.872 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.764+0000 7fc59ba20880 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-13T11:45:42.872 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.764+0000 7fc59ba20880 0 ceph version 16.2.14-456-g445562ab (445562ab4bc3ddfb386936119050695810860bcb) pacific (stable), process ceph-mon, pid 7 2024-01-13T11:45:42.872 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.764+0000 7fc59ba20880 0 pidfile_write: ignore empty --pid-file 2024-01-13T11:45:42.872 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.768+0000 7fc59ba20880 0 load: jerasure load: lrc load: isa 2024-01-13T11:45:42.872 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: RocksDB version: 6.8.1 2024-01-13T11:45:42.872 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-13T11:45:42.872 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Compile date Jan 12 2024 2024-01-13T11:45:42.872 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: DB SUMMARY 2024-01-13T11:45:42.873 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: CURRENT file: CURRENT 2024-01-13T11:45:42.873 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: IDENTITY file: IDENTITY 2024-01-13T11:45:42.873 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-01-13T11:45:42.873 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi088/store.db dir, Total Num: 1, files: 000004.sst 2024-01-13T11:45:42.873 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi088/store.db: 000006.log size: 86533 ; 2024-01-13T11:45:42.873 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.error_if_exists: 0 2024-01-13T11:45:42.873 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.create_if_missing: 0 2024-01-13T11:45:42.873 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.paranoid_checks: 1 2024-01-13T11:45:42.873 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.env: 0x55afc35c3080 2024-01-13T11:45:42.873 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.fs: Posix File System 2024-01-13T11:45:42.873 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.info_log: 0x55afc46f0520 2024-01-13T11:45:42.873 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-13T11:45:42.874 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.statistics: (nil) 2024-01-13T11:45:42.874 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.use_fsync: 0 2024-01-13T11:45:42.874 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.max_log_file_size: 0 2024-01-13T11:45:42.874 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-13T11:45:42.874 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-13T11:45:42.874 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-13T11:45:42.874 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-13T11:45:42.875 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.allow_fallocate: 1 2024-01-13T11:45:42.875 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-13T11:45:42.875 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-13T11:45:42.875 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.use_direct_reads: 0 2024-01-13T11:45:42.875 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-13T11:45:42.875 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.create_missing_column_families: 0 2024-01-13T11:45:42.876 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.db_log_dir: 2024-01-13T11:45:42.876 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi088/store.db 2024-01-13T11:45:42.876 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-13T11:45:42.876 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.max_subcompactions: 1 2024-01-13T11:45:42.876 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.max_background_flushes: -1 2024-01-13T11:45:42.876 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-13T11:45:42.876 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-13T11:45:42.876 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-13T11:45:42.876 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-13T11:45:42.876 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-13T11:45:42.876 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.advise_random_on_open: 1 2024-01-13T11:45:42.876 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-13T11:45:42.876 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.write_buffer_manager: 0x55afc533e480 2024-01-13T11:45:42.876 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-13T11:45:42.877 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-13T11:45:42.877 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-13T11:45:42.877 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-13T11:45:42.877 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.rate_limiter: (nil) 2024-01-13T11:45:42.877 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-13T11:45:42.877 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-13T11:45:42.877 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-13T11:45:42.877 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-13T11:45:42.877 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.unordered_write: 0 2024-01-13T11:45:42.877 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-13T11:45:42.877 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-13T11:45:42.877 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-13T11:45:42.877 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-13T11:45:42.877 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.row_cache: None 2024-01-13T11:45:42.878 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.wal_filter: None 2024-01-13T11:45:42.878 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-13T11:45:42.878 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-13T11:45:42.878 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.preserve_deletes: 0 2024-01-13T11:45:42.878 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.two_write_queues: 0 2024-01-13T11:45:42.878 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.manual_wal_flush: 0 2024-01-13T11:45:42.878 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.atomic_flush: 0 2024-01-13T11:45:42.878 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-13T11:45:42.878 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-13T11:45:42.878 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-13T11:45:42.878 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.log_readahead_size: 0 2024-01-13T11:45:42.878 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-13T11:45:42.878 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.max_background_jobs: 2 2024-01-13T11:45:42.878 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.max_background_compactions: -1 2024-01-13T11:45:42.879 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-13T11:45:42.879 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-13T11:45:42.879 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-13T11:45:42.879 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.max_total_wal_size: 0 2024-01-13T11:45:42.879 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-13T11:45:42.879 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-13T11:45:42.879 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-13T11:45:42.879 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-13T11:45:42.879 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.max_open_files: -1 2024-01-13T11:45:42.879 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.bytes_per_sync: 0 2024-01-13T11:45:42.879 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-13T11:45:42.879 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-13T11:45:42.879 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-13T11:45:42.879 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Compression algorithms supported: 2024-01-13T11:45:42.880 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-13T11:45:42.880 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: kZSTD supported: 0 2024-01-13T11:45:42.880 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: kXpressCompression supported: 0 2024-01-13T11:45:42.880 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-13T11:45:42.880 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: kLZ4Compression supported: 1 2024-01-13T11:45:42.880 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: kBZip2Compression supported: 0 2024-01-13T11:45:42.880 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: kZlibCompression supported: 1 2024-01-13T11:45:42.880 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: kSnappyCompression supported: 1 2024-01-13T11:45:42.880 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-13T11:45:42.880 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi088/store.db/MANIFEST-000005 2024-01-13T11:45:42.880 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-13T11:45:42.880 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-13T11:45:42.880 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.merge_operator: 2024-01-13T11:45:42.881 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.compaction_filter: None 2024-01-13T11:45:42.881 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.compaction_filter_factory: None 2024-01-13T11:45:42.881 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-13T11:45:42.881 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-13T11:45:42.881 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55afc4693d28) 2024-01-13T11:45:42.881 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: cache_index_and_filter_blocks: 1 2024-01-13T11:45:42.881 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-13T11:45:42.881 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-13T11:45:42.881 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: pin_top_level_index_and_filter: 1 2024-01-13T11:45:42.881 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: index_type: 0 2024-01-13T11:45:42.881 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: data_block_index_type: 0 2024-01-13T11:45:42.881 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: index_shortening: 1 2024-01-13T11:45:42.881 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: data_block_hash_table_util_ratio: 0.750000 2024-01-13T11:45:42.882 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: hash_index_allow_collision: 1 2024-01-13T11:45:42.882 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: checksum: 1 2024-01-13T11:45:42.882 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: no_block_cache: 0 2024-01-13T11:45:42.882 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: block_cache: 0x55afc46caf10 2024-01-13T11:45:42.882 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: block_cache_name: BinnedLRUCache 2024-01-13T11:45:42.882 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: block_cache_options: 2024-01-13T11:45:42.882 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: capacity : 536870912 2024-01-13T11:45:42.882 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: num_shard_bits : 4 2024-01-13T11:45:42.882 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: strict_capacity_limit : 0 2024-01-13T11:45:42.882 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: high_pri_pool_ratio: 0.000 2024-01-13T11:45:42.882 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: block_cache_compressed: (nil) 2024-01-13T11:45:42.882 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: persistent_cache: (nil) 2024-01-13T11:45:42.882 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: block_size: 4096 2024-01-13T11:45:42.882 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: block_size_deviation: 10 2024-01-13T11:45:42.882 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: block_restart_interval: 16 2024-01-13T11:45:42.883 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: index_block_restart_interval: 1 2024-01-13T11:45:42.883 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: metadata_block_size: 4096 2024-01-13T11:45:42.883 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: partition_filters: 0 2024-01-13T11:45:42.883 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: use_delta_encoding: 1 2024-01-13T11:45:42.883 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-13T11:45:42.883 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: whole_key_filtering: 1 2024-01-13T11:45:42.883 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: verify_compression: 0 2024-01-13T11:45:42.883 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: read_amp_bytes_per_bit: 0 2024-01-13T11:45:42.883 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: format_version: 2 2024-01-13T11:45:42.883 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: enable_index_compression: 1 2024-01-13T11:45:42.883 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: block_align: 0 2024-01-13T11:45:42.883 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-13T11:45:42.883 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-13T11:45:42.883 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.compression: NoCompression 2024-01-13T11:45:42.884 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-13T11:45:42.884 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-13T11:45:42.884 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-13T11:45:42.884 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.num_levels: 7 2024-01-13T11:45:42.884 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-13T11:45:42.884 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-13T11:45:42.884 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-13T11:45:42.884 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-13T11:45:42.884 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-13T11:45:42.884 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-13T11:45:42.884 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-13T11:45:42.884 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-13T11:45:42.884 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-13T11:45:42.885 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-13T11:45:42.885 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.compression_opts.level: 32767 2024-01-13T11:45:42.885 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-13T11:45:42.885 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-13T11:45:42.885 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-13T11:45:42.885 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.compression_opts.enabled: false 2024-01-13T11:45:42.885 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-13T11:45:42.885 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-13T11:45:42.885 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-13T11:45:42.885 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-13T11:45:42.885 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-13T11:45:42.885 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-13T11:45:42.885 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-13T11:45:42.885 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-13T11:45:42.885 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-13T11:45:42.886 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-13T11:45:42.886 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-13T11:45:42.886 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-13T11:45:42.886 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-13T11:45:42.886 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-13T11:45:42.886 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-13T11:45:42.886 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-13T11:45:42.886 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-13T11:45:42.886 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.arena_block_size: 4194304 2024-01-13T11:45:42.886 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-13T11:45:42.886 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-13T11:45:42.886 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-13T11:45:42.886 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-13T11:45:42.886 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-13T11:45:42.887 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-13T11:45:42.887 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-13T11:45:42.887 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-13T11:45:42.887 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-13T11:45:42.887 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-13T11:45:42.887 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-13T11:45:42.887 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-13T11:45:42.887 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-13T11:45:42.887 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-13T11:45:42.887 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.table_properties_collectors: 2024-01-13T11:45:42.887 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.inplace_update_support: 0 2024-01-13T11:45:42.887 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-13T11:45:42.887 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-13T11:45:42.887 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-13T11:45:42.888 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-13T11:45:42.888 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.bloom_locality: 0 2024-01-13T11:45:42.888 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.max_successive_merges: 0 2024-01-13T11:45:42.888 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-13T11:45:42.888 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-13T11:45:42.888 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.force_consistency_checks: 0 2024-01-13T11:45:42.888 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-13T11:45:42.889 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.ttl: 2592000 2024-01-13T11:45:42.889 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-13T11:45:42.889 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi088/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-13T11:45:42.889 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-01-13T11:45:42.889 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705146342775564, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-01-13T11:45:42.889 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-01-13T11:45:42.889 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-01-13T11:45:42.889 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705146342776534, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 83406, "table_properties": {"data_size": 81673, "index_size": 338, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 581, "raw_key_size": 9331, "raw_average_key_size": 47, "raw_value_size": 76185, "raw_average_value_size": 384, "num_data_blocks": 11, "num_entries": 198, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1705146342, "oldest_key_time": 3, "file_creation_time": 0}} 2024-01-13T11:45:42.889 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-01-13T11:45:42.890 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.772+0000 7fc59ba20880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705146342776962, "job": 1, "event": "recovery_finished"} 2024-01-13T11:45:42.890 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.776+0000 7fc59ba20880 4 rocksdb: DB pointer 0x55afc4793800 2024-01-13T11:45:42.890 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.776+0000 7fc582814700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-13T11:45:42.890 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.776+0000 7fc582814700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-13T11:45:42.890 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: ** DB Stats ** 2024-01-13T11:45:42.890 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T11:45:42.890 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-13T11:45:42.890 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-13T11:45:42.890 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-13T11:45:42.890 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-13T11:45:42.890 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-13T11:45:42.890 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-13T11:45:42.890 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: ** Compaction Stats [default] ** 2024-01-13T11:45:42.890 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T11:45:42.891 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T11:45:42.891 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: L0 2/0 83.14 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 124.9 0.00 0.00 1 0.001 0 0 2024-01-13T11:45:42.891 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: Sum 2/0 83.14 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 124.9 0.00 0.00 1 0.001 0 0 2024-01-13T11:45:42.891 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 124.9 0.00 0.00 1 0.001 0 0 2024-01-13T11:45:42.891 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: ** Compaction Stats [default] ** 2024-01-13T11:45:42.891 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T11:45:42.891 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T11:45:42.891 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 124.9 0.00 0.00 1 0.001 0 0 2024-01-13T11:45:42.891 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T11:45:42.891 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-13T11:45:42.891 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-13T11:45:42.891 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: AddFile(Total Files): cumulative 0, interval 0 2024-01-13T11:45:42.891 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-13T11:45:42.892 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: AddFile(Keys): cumulative 0, interval 0 2024-01-13T11:45:42.892 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: Cumulative compaction: 0.00 GB write, 20.27 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T11:45:42.892 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: Interval compaction: 0.00 GB write, 20.27 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T11:45:42.892 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-13T11:45:42.892 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: ** File Read Latency Histogram By Level [default] ** 2024-01-13T11:45:42.892 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: ** Compaction Stats [default] ** 2024-01-13T11:45:42.892 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T11:45:42.892 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T11:45:42.892 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: L0 2/0 83.14 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 124.9 0.00 0.00 1 0.001 0 0 2024-01-13T11:45:42.892 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: Sum 2/0 83.14 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 124.9 0.00 0.00 1 0.001 0 0 2024-01-13T11:45:42.892 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-13T11:45:42.892 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: ** Compaction Stats [default] ** 2024-01-13T11:45:42.892 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T11:45:42.893 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T11:45:42.893 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 124.9 0.00 0.00 1 0.001 0 0 2024-01-13T11:45:42.893 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T11:45:42.893 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-13T11:45:42.893 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-13T11:45:42.893 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: AddFile(Total Files): cumulative 0, interval 0 2024-01-13T11:45:42.893 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-13T11:45:42.893 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: AddFile(Keys): cumulative 0, interval 0 2024-01-13T11:45:42.893 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: Cumulative compaction: 0.00 GB write, 20.08 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T11:45:42.893 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T11:45:42.893 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-13T11:45:42.893 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: ** File Read Latency Histogram By Level [default] ** 2024-01-13T11:45:42.893 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.776+0000 7fc59ba20880 0 starting mon.smithi088 rank 0 at public addrs [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] at bind addrs [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi088 fsid 236db248-b209-11ee-95ac-87774f69a715 2024-01-13T11:45:42.893 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.776+0000 7fc59ba20880 1 mon.smithi088@-1(???) e1 preinit fsid 236db248-b209-11ee-95ac-87774f69a715 2024-01-13T11:45:42.893 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.776+0000 7fc59ba20880 0 mon.smithi088@-1(???).mds e1 new map 2024-01-13T11:45:42.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.776+0000 7fc59ba20880 0 mon.smithi088@-1(???).mds e1 print_map 2024-01-13T11:45:42.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: e1 2024-01-13T11:45:42.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: enable_multiple, ever_enabled_multiple: 1,1 2024-01-13T11:45:42.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-13T11:45:42.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: legacy client fscid: -1 2024-01-13T11:45:42.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: 2024-01-13T11:45:42.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: No filesystems configured 2024-01-13T11:45:42.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.776+0000 7fc59ba20880 0 mon.smithi088@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-01-13T11:45:42.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.776+0000 7fc59ba20880 0 mon.smithi088@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-13T11:45:42.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.776+0000 7fc59ba20880 0 mon.smithi088@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-13T11:45:42.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.776+0000 7fc59ba20880 0 mon.smithi088@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-13T11:45:42.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: audit 2024-01-13T11:45:39.969382+0000 mon.smithi088 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.88:0/1880630501' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:45:42.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: debug 2024-01-13T11:45:42.776+0000 7fc59ba20880 1 mon.smithi088@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-01-13T11:45:42.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: cluster 2024-01-13T11:45:42.805742+0000 mon.smithi088 (mon.0) 1 : cluster [INF] mon.smithi088 is new leader, mons smithi088 in quorum (ranks 0) 2024-01-13T11:45:42.895 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: cluster 2024-01-13T11:45:42.805825+0000 mon.smithi088 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi088=[v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0]} removed_ranks: {} 2024-01-13T11:45:42.895 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: cluster 2024-01-13T11:45:42.826198+0000 mon.smithi088 (mon.0) 3 : cluster [DBG] fsmap 2024-01-13T11:45:42.895 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: cluster 2024-01-13T11:45:42.826278+0000 mon.smithi088 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-01-13T11:45:42.895 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:42 smithi088 bash[14069]: cluster 2024-01-13T11:45:42.827072+0000 mon.smithi088 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-01-13T11:45:43.617 INFO:teuthology.orchestra.run.smithi088.stderr:Wrote config to /etc/ceph/ceph.conf 2024-01-13T11:45:43.618 INFO:teuthology.orchestra.run.smithi088.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-01-13T11:45:43.618 INFO:teuthology.orchestra.run.smithi088.stderr:Creating mgr... 2024-01-13T11:45:43.618 INFO:teuthology.orchestra.run.smithi088.stderr:Verifying port 9283 ... 2024-01-13T11:45:43.844 INFO:teuthology.orchestra.run.smithi088.stderr:systemctl: Failed to reset failed state of unit ceph-236db248-b209-11ee-95ac-87774f69a715@mgr.smithi088.xwlxdh.service: Unit ceph-236db248-b209-11ee-95ac-87774f69a715@mgr.smithi088.xwlxdh.service not loaded. 2024-01-13T11:45:43.853 INFO:teuthology.orchestra.run.smithi088.stderr:systemctl: Created symlink /etc/systemd/system/ceph-236db248-b209-11ee-95ac-87774f69a715.target.wants/ceph-236db248-b209-11ee-95ac-87774f69a715@mgr.smithi088.xwlxdh.service -> /etc/systemd/system/ceph-236db248-b209-11ee-95ac-87774f69a715@.service. 2024-01-13T11:45:44.068 INFO:teuthology.orchestra.run.smithi088.stderr:firewalld does not appear to be present 2024-01-13T11:45:44.068 INFO:teuthology.orchestra.run.smithi088.stderr:Not possible to enable service . firewalld.service is not available 2024-01-13T11:45:44.068 INFO:teuthology.orchestra.run.smithi088.stderr:firewalld does not appear to be present 2024-01-13T11:45:44.068 INFO:teuthology.orchestra.run.smithi088.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-01-13T11:45:44.068 INFO:teuthology.orchestra.run.smithi088.stderr:Waiting for mgr to start... 2024-01-13T11:45:44.068 INFO:teuthology.orchestra.run.smithi088.stderr:Waiting for mgr... 2024-01-13T11:45:44.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:43 smithi088 bash[14069]: audit 2024-01-13T11:45:42.973612+0000 mon.smithi088 (mon.0) 6 : audit [INF] from='client.? 172.21.15.88:0/2990556829' entity='client.admin' 2024-01-13T11:45:45.549 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: 2024-01-13T11:45:45.549 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: { 2024-01-13T11:45:45.549 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "fsid": "236db248-b209-11ee-95ac-87774f69a715", 2024-01-13T11:45:45.549 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "health": { 2024-01-13T11:45:45.549 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-13T11:45:45.550 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "checks": {}, 2024-01-13T11:45:45.550 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "mutes": [] 2024-01-13T11:45:45.550 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: }, 2024-01-13T11:45:45.551 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-13T11:45:45.551 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "quorum": [ 2024-01-13T11:45:45.551 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: 0 2024-01-13T11:45:45.551 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: ], 2024-01-13T11:45:45.551 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-13T11:45:45.552 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "smithi088" 2024-01-13T11:45:45.552 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: ], 2024-01-13T11:45:45.553 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-01-13T11:45:45.553 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "monmap": { 2024-01-13T11:45:45.553 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T11:45:45.553 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-13T11:45:45.553 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-13T11:45:45.553 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: }, 2024-01-13T11:45:45.553 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "osdmap": { 2024-01-13T11:45:45.553 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T11:45:45.557 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-13T11:45:45.557 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-13T11:45:45.558 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-13T11:45:45.558 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-13T11:45:45.558 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-13T11:45:45.558 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-13T11:45:45.558 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: }, 2024-01-13T11:45:45.558 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "pgmap": { 2024-01-13T11:45:45.558 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-13T11:45:45.558 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-13T11:45:45.558 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-13T11:45:45.558 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-13T11:45:45.558 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-13T11:45:45.558 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-13T11:45:45.558 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-13T11:45:45.558 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-13T11:45:45.559 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: }, 2024-01-13T11:45:45.559 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "fsmap": { 2024-01-13T11:45:45.559 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T11:45:45.559 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-13T11:45:45.559 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-13T11:45:45.559 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: }, 2024-01-13T11:45:45.559 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-13T11:45:45.559 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "available": false, 2024-01-13T11:45:45.559 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-13T11:45:45.559 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "modules": [ 2024-01-13T11:45:45.559 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "iostat", 2024-01-13T11:45:45.559 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "nfs", 2024-01-13T11:45:45.559 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "restful" 2024-01-13T11:45:45.559 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: ], 2024-01-13T11:45:45.559 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "services": {} 2024-01-13T11:45:45.560 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: }, 2024-01-13T11:45:45.560 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "servicemap": { 2024-01-13T11:45:45.560 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T11:45:45.560 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "modified": "2024-01-13T11:45:36.236749+0000", 2024-01-13T11:45:45.560 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "services": {} 2024-01-13T11:45:45.560 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: }, 2024-01-13T11:45:45.560 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-13T11:45:45.560 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: } 2024-01-13T11:45:45.795 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:45 smithi088 bash[14069]: audit 2024-01-13T11:45:45.555633+0000 mon.smithi088 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.88:0/2779514523' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-13T11:45:46.248 INFO:teuthology.orchestra.run.smithi088.stderr:mgr not available, waiting (1/15)... 2024-01-13T11:45:49.435 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: 2024-01-13T11:45:49.435 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: { 2024-01-13T11:45:49.435 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "fsid": "236db248-b209-11ee-95ac-87774f69a715", 2024-01-13T11:45:49.435 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "health": { 2024-01-13T11:45:49.436 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-13T11:45:49.436 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "checks": {}, 2024-01-13T11:45:49.436 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "mutes": [] 2024-01-13T11:45:49.436 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: }, 2024-01-13T11:45:49.436 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-13T11:45:49.436 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "quorum": [ 2024-01-13T11:45:49.436 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: 0 2024-01-13T11:45:49.436 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: ], 2024-01-13T11:45:49.436 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-13T11:45:49.438 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "smithi088" 2024-01-13T11:45:49.438 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: ], 2024-01-13T11:45:49.438 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "quorum_age": 6, 2024-01-13T11:45:49.438 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "monmap": { 2024-01-13T11:45:49.438 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T11:45:49.438 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-13T11:45:49.438 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-13T11:45:49.438 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: }, 2024-01-13T11:45:49.439 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "osdmap": { 2024-01-13T11:45:49.439 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T11:45:49.439 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-13T11:45:49.439 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-13T11:45:49.439 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-13T11:45:49.439 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-13T11:45:49.443 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-13T11:45:49.443 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-13T11:45:49.443 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: }, 2024-01-13T11:45:49.443 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "pgmap": { 2024-01-13T11:45:49.443 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-13T11:45:49.443 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-13T11:45:49.443 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-13T11:45:49.443 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-13T11:45:49.443 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-13T11:45:49.443 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-13T11:45:49.443 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-13T11:45:49.443 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-13T11:45:49.443 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: }, 2024-01-13T11:45:49.443 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "fsmap": { 2024-01-13T11:45:49.444 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T11:45:49.444 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-13T11:45:49.444 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-13T11:45:49.444 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: }, 2024-01-13T11:45:49.444 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-13T11:45:49.444 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "available": false, 2024-01-13T11:45:49.444 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-13T11:45:49.444 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "modules": [ 2024-01-13T11:45:49.444 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "iostat", 2024-01-13T11:45:49.444 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "nfs", 2024-01-13T11:45:49.444 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "restful" 2024-01-13T11:45:49.444 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: ], 2024-01-13T11:45:49.444 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "services": {} 2024-01-13T11:45:49.444 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: }, 2024-01-13T11:45:49.445 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "servicemap": { 2024-01-13T11:45:49.445 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T11:45:49.445 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "modified": "2024-01-13T11:45:36.236749+0000", 2024-01-13T11:45:49.445 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "services": {} 2024-01-13T11:45:49.445 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: }, 2024-01-13T11:45:49.445 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-13T11:45:49.445 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: } 2024-01-13T11:45:49.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:49 smithi088 bash[14069]: audit 2024-01-13T11:45:49.440612+0000 mon.smithi088 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.88:0/3995515648' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-13T11:45:50.097 INFO:teuthology.orchestra.run.smithi088.stderr:mgr not available, waiting (2/15)... 2024-01-13T11:45:52.131 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:51 smithi088 bash[14069]: cluster 2024-01-13T11:45:51.763662+0000 mon.smithi088 (mon.0) 9 : cluster [INF] Activating manager daemon smithi088.xwlxdh 2024-01-13T11:45:52.131 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:51 smithi088 bash[14069]: cluster 2024-01-13T11:45:51.769531+0000 mon.smithi088 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi088.xwlxdh(active, starting, since 0.00608266s) 2024-01-13T11:45:52.131 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:51 smithi088 bash[14069]: audit 2024-01-13T11:45:51.772214+0000 mon.smithi088 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.88:0/1231514363' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-13T11:45:52.132 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:51 smithi088 bash[14069]: audit 2024-01-13T11:45:51.772774+0000 mon.smithi088 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.88:0/1231514363' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-13T11:45:52.132 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:51 smithi088 bash[14069]: audit 2024-01-13T11:45:51.773365+0000 mon.smithi088 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.88:0/1231514363' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-13T11:45:52.132 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:51 smithi088 bash[14069]: audit 2024-01-13T11:45:51.773788+0000 mon.smithi088 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.88:0/1231514363' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mon metadata", "id": "smithi088"}]: dispatch 2024-01-13T11:45:52.132 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:51 smithi088 bash[14069]: audit 2024-01-13T11:45:51.774097+0000 mon.smithi088 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.88:0/1231514363' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mgr metadata", "who": "smithi088.xwlxdh", "id": "smithi088.xwlxdh"}]: dispatch 2024-01-13T11:45:52.132 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:51 smithi088 bash[14069]: cluster 2024-01-13T11:45:51.797929+0000 mon.smithi088 (mon.0) 16 : cluster [INF] Manager daemon smithi088.xwlxdh is now available 2024-01-13T11:45:52.132 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:51 smithi088 bash[14069]: audit 2024-01-13T11:45:51.815462+0000 mon.smithi088 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.88:0/1231514363' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:45:53.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:52 smithi088 bash[14069]: audit 2024-01-13T11:45:51.825429+0000 mon.smithi088 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.88:0/1231514363' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi088.xwlxdh/mirror_snapshot_schedule"}]: dispatch 2024-01-13T11:45:53.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:52 smithi088 bash[14069]: audit 2024-01-13T11:45:51.831858+0000 mon.smithi088 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.88:0/1231514363' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:45:53.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:52 smithi088 bash[14069]: audit 2024-01-13T11:45:51.832052+0000 mon.smithi088 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.88:0/1231514363' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi088.xwlxdh/trash_purge_schedule"}]: dispatch 2024-01-13T11:45:53.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:52 smithi088 bash[14069]: audit 2024-01-13T11:45:51.835255+0000 mon.smithi088 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.88:0/1231514363' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:45:53.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:52 smithi088 bash[14069]: cluster 2024-01-13T11:45:52.774858+0000 mon.smithi088 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi088.xwlxdh(active, since 1.01141s) 2024-01-13T11:45:53.577 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: 2024-01-13T11:45:53.577 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: { 2024-01-13T11:45:53.577 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "fsid": "236db248-b209-11ee-95ac-87774f69a715", 2024-01-13T11:45:53.578 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "health": { 2024-01-13T11:45:53.579 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-13T11:45:53.579 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "checks": {}, 2024-01-13T11:45:53.579 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "mutes": [] 2024-01-13T11:45:53.579 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: }, 2024-01-13T11:45:53.579 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-13T11:45:53.579 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "quorum": [ 2024-01-13T11:45:53.579 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: 0 2024-01-13T11:45:53.579 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: ], 2024-01-13T11:45:53.579 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-13T11:45:53.580 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "smithi088" 2024-01-13T11:45:53.580 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: ], 2024-01-13T11:45:53.580 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "quorum_age": 10, 2024-01-13T11:45:53.581 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "monmap": { 2024-01-13T11:45:53.581 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T11:45:53.581 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-13T11:45:53.581 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-13T11:45:53.581 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: }, 2024-01-13T11:45:53.581 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "osdmap": { 2024-01-13T11:45:53.581 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T11:45:53.581 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-13T11:45:53.581 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-13T11:45:53.581 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-13T11:45:53.585 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-13T11:45:53.585 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-13T11:45:53.585 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-13T11:45:53.586 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: }, 2024-01-13T11:45:53.586 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "pgmap": { 2024-01-13T11:45:53.586 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-13T11:45:53.586 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-13T11:45:53.586 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-13T11:45:53.586 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-13T11:45:53.586 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-13T11:45:53.586 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-13T11:45:53.586 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-13T11:45:53.586 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-13T11:45:53.586 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: }, 2024-01-13T11:45:53.586 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "fsmap": { 2024-01-13T11:45:53.586 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T11:45:53.586 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-13T11:45:53.587 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-13T11:45:53.587 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: }, 2024-01-13T11:45:53.587 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-13T11:45:53.587 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "available": true, 2024-01-13T11:45:53.587 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-13T11:45:53.587 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "modules": [ 2024-01-13T11:45:53.587 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "iostat", 2024-01-13T11:45:53.587 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "nfs", 2024-01-13T11:45:53.587 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "restful" 2024-01-13T11:45:53.587 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: ], 2024-01-13T11:45:53.587 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "services": {} 2024-01-13T11:45:53.587 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: }, 2024-01-13T11:45:53.587 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "servicemap": { 2024-01-13T11:45:53.587 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T11:45:53.587 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "modified": "2024-01-13T11:45:36.236749+0000", 2024-01-13T11:45:53.588 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "services": {} 2024-01-13T11:45:53.588 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: }, 2024-01-13T11:45:53.588 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-13T11:45:53.588 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: } 2024-01-13T11:45:53.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:53 smithi088 bash[14069]: audit 2024-01-13T11:45:53.584017+0000 mon.smithi088 (mon.0) 23 : audit [DBG] from='client.? 172.21.15.88:0/4164320571' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-13T11:45:54.234 INFO:teuthology.orchestra.run.smithi088.stderr:mgr is available 2024-01-13T11:45:55.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:54 smithi088 bash[14069]: cluster 2024-01-13T11:45:53.825073+0000 mon.smithi088 (mon.0) 24 : cluster [DBG] mgrmap e4: smithi088.xwlxdh(active, since 2s) 2024-01-13T11:45:55.514 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: 2024-01-13T11:45:55.514 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: [global] 2024-01-13T11:45:55.514 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: fsid = 236db248-b209-11ee-95ac-87774f69a715 2024-01-13T11:45:55.514 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-01-13T11:45:55.515 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-01-13T11:45:55.515 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-01-13T11:45:55.515 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-01-13T11:45:55.516 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: 2024-01-13T11:45:55.516 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: [mgr] 2024-01-13T11:45:55.516 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-01-13T11:45:55.516 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: 2024-01-13T11:45:55.516 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: [osd] 2024-01-13T11:45:55.516 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-01-13T11:45:55.516 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-01-13T11:45:56.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:55 smithi088 bash[14069]: audit 2024-01-13T11:45:55.519798+0000 mon.smithi088 (mon.0) 25 : audit [INF] from='client.? 172.21.15.88:0/333513086' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-13T11:45:56.183 INFO:teuthology.orchestra.run.smithi088.stderr:Enabling cephadm module... 2024-01-13T11:45:57.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:57 smithi088 bash[14069]: audit 2024-01-13T11:45:57.568078+0000 mon.smithi088 (mon.0) 26 : audit [INF] from='client.? 172.21.15.88:0/1637885793' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-01-13T11:45:58.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:58 smithi088 bash[14069]: audit 2024-01-13T11:45:57.624680+0000 mon.smithi088 (mon.0) 27 : audit [INF] from='client.? 172.21.15.88:0/1637885793' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-01-13T11:45:58.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:58 smithi088 bash[14069]: cluster 2024-01-13T11:45:57.624867+0000 mon.smithi088 (mon.0) 28 : cluster [DBG] mgrmap e5: smithi088.xwlxdh(active, since 5s) 2024-01-13T11:45:59.735 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: { 2024-01-13T11:45:59.735 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "epoch": 5, 2024-01-13T11:45:59.735 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "available": true, 2024-01-13T11:45:59.735 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "active_name": "smithi088.xwlxdh", 2024-01-13T11:45:59.735 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "num_standby": 0 2024-01-13T11:45:59.735 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: } 2024-01-13T11:45:59.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:45:59 smithi088 bash[14069]: audit 2024-01-13T11:45:59.741711+0000 mon.smithi088 (mon.0) 29 : audit [DBG] from='client.? 172.21.15.88:0/2961658943' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-13T11:46:00.391 INFO:teuthology.orchestra.run.smithi088.stderr:Waiting for the mgr to restart... 2024-01-13T11:46:00.391 INFO:teuthology.orchestra.run.smithi088.stderr:Waiting for mgr epoch 5... 2024-01-13T11:46:04.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:04 smithi088 bash[14069]: cluster 2024-01-13T11:46:04.119351+0000 mon.smithi088 (mon.0) 30 : cluster [INF] Active manager daemon smithi088.xwlxdh restarted 2024-01-13T11:46:04.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:04 smithi088 bash[14069]: cluster 2024-01-13T11:46:04.119794+0000 mon.smithi088 (mon.0) 31 : cluster [INF] Activating manager daemon smithi088.xwlxdh 2024-01-13T11:46:04.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:04 smithi088 bash[14069]: cluster 2024-01-13T11:46:04.125525+0000 mon.smithi088 (mon.0) 32 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-01-13T11:46:04.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:04 smithi088 bash[14069]: cluster 2024-01-13T11:46:04.125744+0000 mon.smithi088 (mon.0) 33 : cluster [DBG] mgrmap e6: smithi088.xwlxdh(active, starting, since 0.00609809s) 2024-01-13T11:46:04.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:04 smithi088 bash[14069]: audit 2024-01-13T11:46:04.127759+0000 mon.smithi088 (mon.0) 34 : audit [DBG] from='mgr.14116 172.21.15.88:0/3325659183' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mon metadata", "id": "smithi088"}]: dispatch 2024-01-13T11:46:04.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:04 smithi088 bash[14069]: audit 2024-01-13T11:46:04.128623+0000 mon.smithi088 (mon.0) 35 : audit [DBG] from='mgr.14116 172.21.15.88:0/3325659183' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mgr metadata", "who": "smithi088.xwlxdh", "id": "smithi088.xwlxdh"}]: dispatch 2024-01-13T11:46:04.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:04 smithi088 bash[14069]: audit 2024-01-13T11:46:04.129563+0000 mon.smithi088 (mon.0) 36 : audit [DBG] from='mgr.14116 172.21.15.88:0/3325659183' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-13T11:46:04.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:04 smithi088 bash[14069]: audit 2024-01-13T11:46:04.129726+0000 mon.smithi088 (mon.0) 37 : audit [DBG] from='mgr.14116 172.21.15.88:0/3325659183' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-13T11:46:04.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:04 smithi088 bash[14069]: audit 2024-01-13T11:46:04.129872+0000 mon.smithi088 (mon.0) 38 : audit [DBG] from='mgr.14116 172.21.15.88:0/3325659183' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-13T11:46:04.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:04 smithi088 bash[14069]: cluster 2024-01-13T11:46:04.154103+0000 mon.smithi088 (mon.0) 39 : cluster [INF] Manager daemon smithi088.xwlxdh is now available 2024-01-13T11:46:04.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:04 smithi088 bash[14069]: audit 2024-01-13T11:46:04.175016+0000 mon.smithi088 (mon.0) 40 : audit [INF] from='mgr.14116 172.21.15.88:0/3325659183' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:46:05.137 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: { 2024-01-13T11:46:05.137 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-01-13T11:46:05.137 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "initialized": true 2024-01-13T11:46:05.137 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: } 2024-01-13T11:46:05.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:05 smithi088 bash[14069]: cephadm 2024-01-13T11:46:04.169277+0000 mgr.smithi088.xwlxdh (mgr.14116) 1 : cephadm [INF] Found migration_current of "None". Setting to last migration. 2024-01-13T11:46:05.645 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:05 smithi088 bash[14069]: audit 2024-01-13T11:46:04.290148+0000 mon.smithi088 (mon.0) 41 : audit [INF] from='mgr.14116 172.21.15.88:0/3325659183' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:46:05.645 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:05 smithi088 bash[14069]: audit 2024-01-13T11:46:04.291794+0000 mon.smithi088 (mon.0) 42 : audit [DBG] from='mgr.14116 172.21.15.88:0/3325659183' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:46:05.645 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:05 smithi088 bash[14069]: audit 2024-01-13T11:46:04.293836+0000 mon.smithi088 (mon.0) 43 : audit [DBG] from='mgr.14116 172.21.15.88:0/3325659183' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:46:05.645 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:05 smithi088 bash[14069]: audit 2024-01-13T11:46:04.297264+0000 mon.smithi088 (mon.0) 44 : audit [DBG] from='mgr.14116 172.21.15.88:0/3325659183' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:46:05.645 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:05 smithi088 bash[14069]: audit 2024-01-13T11:46:04.320990+0000 mon.smithi088 (mon.0) 45 : audit [INF] from='mgr.14116 172.21.15.88:0/3325659183' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi088.xwlxdh/mirror_snapshot_schedule"}]: dispatch 2024-01-13T11:46:05.645 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:05 smithi088 bash[14069]: audit 2024-01-13T11:46:04.325434+0000 mon.smithi088 (mon.0) 46 : audit [INF] from='mgr.14116 172.21.15.88:0/3325659183' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi088.xwlxdh/trash_purge_schedule"}]: dispatch 2024-01-13T11:46:05.645 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:05 smithi088 bash[14069]: cluster 2024-01-13T11:46:05.130855+0000 mon.smithi088 (mon.0) 47 : cluster [DBG] mgrmap e7: smithi088.xwlxdh(active, since 1.0112s) 2024-01-13T11:46:05.830 INFO:teuthology.orchestra.run.smithi088.stderr:mgr epoch 5 is available 2024-01-13T11:46:05.830 INFO:teuthology.orchestra.run.smithi088.stderr:Setting orchestrator backend to cephadm... 2024-01-13T11:46:06.543 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:06 smithi088 bash[14069]: audit 2024-01-13T11:46:05.128882+0000 mgr.smithi088.xwlxdh (mgr.14116) 2 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-13T11:46:06.543 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:06 smithi088 bash[14069]: audit 2024-01-13T11:46:05.144191+0000 mgr.smithi088.xwlxdh (mgr.14116) 3 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-13T11:46:07.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:07 smithi088 bash[14069]: cluster 2024-01-13T11:46:06.295655+0000 mon.smithi088 (mon.0) 48 : cluster [DBG] mgrmap e8: smithi088.xwlxdh(active, since 2s) 2024-01-13T11:46:07.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:07 smithi088 bash[14069]: audit 2024-01-13T11:46:07.138182+0000 mon.smithi088 (mon.0) 49 : audit [INF] from='mgr.14116 172.21.15.88:0/3325659183' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:46:07.645 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:07 smithi088 bash[14069]: audit 2024-01-13T11:46:07.178517+0000 mon.smithi088 (mon.0) 50 : audit [DBG] from='mgr.14116 172.21.15.88:0/3325659183' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:46:08.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:08 smithi088 bash[14069]: audit 2024-01-13T11:46:07.128562+0000 mgr.smithi088.xwlxdh (mgr.14116) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T11:46:09.004 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: value unchanged 2024-01-13T11:46:09.618 INFO:teuthology.orchestra.run.smithi088.stderr:Generating ssh key... 2024-01-13T11:46:10.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:09 smithi088 bash[14069]: audit 2024-01-13T11:46:09.010993+0000 mgr.smithi088.xwlxdh (mgr.14116) 5 : audit [DBG] from='client.14128 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T11:46:12.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:12 smithi088 bash[14069]: audit 2024-01-13T11:46:10.875109+0000 mgr.smithi088.xwlxdh (mgr.14116) 6 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T11:46:12.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:12 smithi088 bash[14069]: cephadm 2024-01-13T11:46:10.875548+0000 mgr.smithi088.xwlxdh (mgr.14116) 7 : cephadm [INF] Generating ssh key... 2024-01-13T11:46:12.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:12 smithi088 bash[14069]: audit 2024-01-13T11:46:11.333848+0000 mon.smithi088 (mon.0) 51 : audit [INF] from='mgr.14116 172.21.15.88:0/3325659183' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:46:12.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:12 smithi088 bash[14069]: audit 2024-01-13T11:46:11.337853+0000 mon.smithi088 (mon.0) 52 : audit [INF] from='mgr.14116 172.21.15.88:0/3325659183' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:46:13.251 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDrdl3yroTiZdGfEf9MssfN62V1BN1jftfPO7qihpK7W3au9489FxkX8lvDgihOw/QaEx4j6jNsEFL494tw/kp1oMCEUUMW26j9fe5Peb1R9HsntdJvxAnTRys6ipMJOhs4fO0PdAC5kxVSOpGicwsVkKrtwNpxmToHB38HYHniG2OTvwJ0U7ei1SF0k7vSRxHJVNw4bL7mMG6Cy2cvFDbUNRQkWD5t6DDNhlbjbFM9cFw8tH8WA2nyVl30yFZmtBhoojBGCA8no374MTeDeI5yjwh+qpmj7bo59zKExmPxS71mf+/VshPefkdUQar/aCMSB/W2I/IsjAP1SKRFpbmJhe2jVR3714cimZ8LF89RRFiRbp1HAQG+vBmH08X6mc3sgL8zGSy3yj6tw2IpIKK8bcj/Zerg2hupKWQXs02pWWJX7kluYTHuDgTmqKema49Rv4YDDeOlfb55X78Zh2Inn2yU6Jb+OflD3enSangD+hHcUISjaug14NiQ3lkszdE= ceph-236db248-b209-11ee-95ac-87774f69a715 2024-01-13T11:46:13.846 INFO:teuthology.orchestra.run.smithi088.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-01-13T11:46:13.846 INFO:teuthology.orchestra.run.smithi088.stderr:Adding key to root@localhost authorized_keys... 2024-01-13T11:46:13.846 INFO:teuthology.orchestra.run.smithi088.stderr:Adding host smithi088... 2024-01-13T11:46:14.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:13 smithi088 bash[14069]: audit 2024-01-13T11:46:13.257930+0000 mgr.smithi088.xwlxdh (mgr.14116) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T11:46:16.043 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: Added host 'smithi088' with addr '172.21.15.88' 2024-01-13T11:46:16.100 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:15 smithi088 bash[14069]: audit 2024-01-13T11:46:15.207551+0000 mgr.smithi088.xwlxdh (mgr.14116) 9 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi088", "addr": "172.21.15.88", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T11:46:16.967 INFO:teuthology.orchestra.run.smithi088.stderr:Deploying mon service with default placement... 2024-01-13T11:46:17.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:17 smithi088 bash[14069]: audit 2024-01-13T11:46:16.049208+0000 mon.smithi088 (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.88:0/3325659183' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:46:17.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:17 smithi088 bash[14069]: cephadm 2024-01-13T11:46:16.049750+0000 mgr.smithi088.xwlxdh (mgr.14116) 10 : cephadm [INF] Added host smithi088 2024-01-13T11:46:17.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:17 smithi088 bash[14069]: audit 2024-01-13T11:46:16.083533+0000 mon.smithi088 (mon.0) 54 : audit [DBG] from='mgr.14116 172.21.15.88:0/3325659183' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:46:19.062 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: Scheduled mon update... 2024-01-13T11:46:19.878 INFO:teuthology.orchestra.run.smithi088.stderr:Deploying mgr service with default placement... 2024-01-13T11:46:20.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:20 smithi088 bash[14069]: audit 2024-01-13T11:46:19.063053+0000 mgr.smithi088.xwlxdh (mgr.14116) 11 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T11:46:20.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:20 smithi088 bash[14069]: cephadm 2024-01-13T11:46:19.064479+0000 mgr.smithi088.xwlxdh (mgr.14116) 12 : cephadm [INF] Saving service mon spec with placement count:5 2024-01-13T11:46:20.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:20 smithi088 bash[14069]: audit 2024-01-13T11:46:19.068852+0000 mon.smithi088 (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.88:0/3325659183' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:46:20.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:20 smithi088 bash[14069]: audit 2024-01-13T11:46:19.539591+0000 mon.smithi088 (mon.0) 56 : audit [INF] from='mgr.14116 172.21.15.88:0/3325659183' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:46:20.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:20 smithi088 bash[14069]: audit 2024-01-13T11:46:19.785543+0000 mon.smithi088 (mon.0) 57 : audit [INF] from='mgr.14116 172.21.15.88:0/3325659183' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:46:21.217 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-01-13T11:46:21.949 INFO:teuthology.orchestra.run.smithi088.stderr:Deploying crash service with default placement... 2024-01-13T11:46:22.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:22 smithi088 bash[14069]: audit 2024-01-13T11:46:21.219343+0000 mgr.smithi088.xwlxdh (mgr.14116) 13 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T11:46:22.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:22 smithi088 bash[14069]: cephadm 2024-01-13T11:46:21.220411+0000 mgr.smithi088.xwlxdh (mgr.14116) 14 : cephadm [INF] Saving service mgr spec with placement count:2 2024-01-13T11:46:22.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:22 smithi088 bash[14069]: audit 2024-01-13T11:46:21.224102+0000 mon.smithi088 (mon.0) 58 : audit [INF] from='mgr.14116 172.21.15.88:0/3325659183' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:46:23.302 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: Scheduled crash update... 2024-01-13T11:46:23.991 INFO:teuthology.orchestra.run.smithi088.stderr:Deploying prometheus service with default placement... 2024-01-13T11:46:24.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:24 smithi088 bash[14069]: audit 2024-01-13T11:46:23.303666+0000 mgr.smithi088.xwlxdh (mgr.14116) 15 : audit [DBG] from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T11:46:24.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:24 smithi088 bash[14069]: cephadm 2024-01-13T11:46:23.304852+0000 mgr.smithi088.xwlxdh (mgr.14116) 16 : cephadm [INF] Saving service crash spec with placement * 2024-01-13T11:46:24.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:24 smithi088 bash[14069]: audit 2024-01-13T11:46:23.309177+0000 mon.smithi088 (mon.0) 59 : audit [INF] from='mgr.14116 172.21.15.88:0/3325659183' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:46:25.350 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-01-13T11:46:25.611 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:25 smithi088 bash[14069]: cluster 2024-01-13T11:46:24.131583+0000 mgr.smithi088.xwlxdh (mgr.14116) 17 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:46:25.612 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:25 smithi088 bash[14069]: audit 2024-01-13T11:46:24.795340+0000 mon.smithi088 (mon.0) 60 : audit [INF] from='mgr.14116 172.21.15.88:0/3325659183' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:46:26.071 INFO:teuthology.orchestra.run.smithi088.stderr:Deploying grafana service with default placement... 2024-01-13T11:46:26.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:26 smithi088 bash[14069]: audit 2024-01-13T11:46:25.351690+0000 mgr.smithi088.xwlxdh (mgr.14116) 18 : audit [DBG] from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T11:46:26.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:26 smithi088 bash[14069]: cephadm 2024-01-13T11:46:25.353891+0000 mgr.smithi088.xwlxdh (mgr.14116) 19 : cephadm [INF] Saving service prometheus spec with placement count:1 2024-01-13T11:46:26.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:26 smithi088 bash[14069]: audit 2024-01-13T11:46:25.357365+0000 mon.smithi088 (mon.0) 61 : audit [INF] from='mgr.14116 172.21.15.88:0/3325659183' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:46:27.418 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-01-13T11:46:27.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:27 smithi088 bash[14069]: cluster 2024-01-13T11:46:26.131989+0000 mgr.smithi088.xwlxdh (mgr.14116) 20 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:46:28.172 INFO:teuthology.orchestra.run.smithi088.stderr:Deploying node-exporter service with default placement... 2024-01-13T11:46:28.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:28 smithi088 bash[14069]: audit 2024-01-13T11:46:27.418095+0000 mgr.smithi088.xwlxdh (mgr.14116) 21 : audit [DBG] from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T11:46:28.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:28 smithi088 bash[14069]: cephadm 2024-01-13T11:46:27.420197+0000 mgr.smithi088.xwlxdh (mgr.14116) 22 : cephadm [INF] Saving service grafana spec with placement count:1 2024-01-13T11:46:28.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:28 smithi088 bash[14069]: audit 2024-01-13T11:46:27.424771+0000 mon.smithi088 (mon.0) 62 : audit [INF] from='mgr.14116 172.21.15.88:0/3325659183' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:46:29.711 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:29 smithi088 bash[14069]: cluster 2024-01-13T11:46:28.132432+0000 mgr.smithi088.xwlxdh (mgr.14116) 23 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:46:29.992 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-01-13T11:46:31.026 INFO:teuthology.orchestra.run.smithi088.stderr:Deploying alertmanager service with default placement... 2024-01-13T11:46:31.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:30 smithi088 bash[14069]: audit 2024-01-13T11:46:29.995846+0000 mgr.smithi088.xwlxdh (mgr.14116) 24 : audit [DBG] from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T11:46:31.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:30 smithi088 bash[14069]: cephadm 2024-01-13T11:46:29.996898+0000 mgr.smithi088.xwlxdh (mgr.14116) 25 : cephadm [INF] Saving service node-exporter spec with placement * 2024-01-13T11:46:31.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:30 smithi088 bash[14069]: audit 2024-01-13T11:46:29.999203+0000 mon.smithi088 (mon.0) 63 : audit [INF] from='mgr.14116 172.21.15.88:0/3325659183' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:46:31.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:30 smithi088 bash[14069]: cluster 2024-01-13T11:46:30.132819+0000 mgr.smithi088.xwlxdh (mgr.14116) 26 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:46:32.426 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-01-13T11:46:33.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:33 smithi088 bash[14069]: cluster 2024-01-13T11:46:32.133294+0000 mgr.smithi088.xwlxdh (mgr.14116) 27 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:46:33.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:33 smithi088 bash[14069]: audit 2024-01-13T11:46:32.426285+0000 mgr.smithi088.xwlxdh (mgr.14116) 28 : audit [DBG] from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T11:46:33.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:33 smithi088 bash[14069]: cephadm 2024-01-13T11:46:32.428410+0000 mgr.smithi088.xwlxdh (mgr.14116) 29 : cephadm [INF] Saving service alertmanager spec with placement count:1 2024-01-13T11:46:33.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:33 smithi088 bash[14069]: audit 2024-01-13T11:46:32.433129+0000 mon.smithi088 (mon.0) 64 : audit [INF] from='mgr.14116 172.21.15.88:0/3325659183' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:46:35.819 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:35 smithi088 bash[14069]: cluster 2024-01-13T11:46:34.133595+0000 mgr.smithi088.xwlxdh (mgr.14116) 30 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:46:35.819 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:35 smithi088 bash[14069]: audit 2024-01-13T11:46:34.430914+0000 mon.smithi088 (mon.0) 65 : audit [INF] from='client.? 172.21.15.88:0/3651406864' entity='client.admin' 2024-01-13T11:46:37.074 INFO:teuthology.orchestra.run.smithi088.stderr:Enabling the dashboard module... 2024-01-13T11:46:37.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:37 smithi088 bash[14069]: cluster 2024-01-13T11:46:36.134047+0000 mgr.smithi088.xwlxdh (mgr.14116) 31 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:46:37.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:37 smithi088 bash[14069]: audit 2024-01-13T11:46:36.397159+0000 mon.smithi088 (mon.0) 66 : audit [INF] from='client.? 172.21.15.88:0/1367335815' entity='client.admin' 2024-01-13T11:46:39.111 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:38 smithi088 bash[14069]: cluster 2024-01-13T11:46:38.134467+0000 mgr.smithi088.xwlxdh (mgr.14116) 32 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:46:39.112 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:38 smithi088 bash[14069]: audit 2024-01-13T11:46:38.477430+0000 mon.smithi088 (mon.0) 67 : audit [INF] from='client.? 172.21.15.88:0/443550667' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-01-13T11:46:40.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:39 smithi088 bash[14069]: audit 2024-01-13T11:46:38.804962+0000 mon.smithi088 (mon.0) 68 : audit [INF] from='client.? 172.21.15.88:0/443550667' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-01-13T11:46:40.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:39 smithi088 bash[14069]: cluster 2024-01-13T11:46:38.805066+0000 mon.smithi088 (mon.0) 69 : cluster [DBG] mgrmap e9: smithi088.xwlxdh(active, since 34s) 2024-01-13T11:46:40.984 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: { 2024-01-13T11:46:40.984 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "epoch": 9, 2024-01-13T11:46:40.984 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "available": true, 2024-01-13T11:46:40.984 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "active_name": "smithi088.xwlxdh", 2024-01-13T11:46:40.984 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "num_standby": 0 2024-01-13T11:46:40.984 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: } 2024-01-13T11:46:41.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:41 smithi088 bash[14069]: audit 2024-01-13T11:46:40.990905+0000 mon.smithi088 (mon.0) 70 : audit [DBG] from='client.? 172.21.15.88:0/526198838' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-13T11:46:41.651 INFO:teuthology.orchestra.run.smithi088.stderr:Waiting for the mgr to restart... 2024-01-13T11:46:41.651 INFO:teuthology.orchestra.run.smithi088.stderr:Waiting for mgr epoch 9... 2024-01-13T11:46:45.761 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:45 smithi088 bash[14069]: cluster 2024-01-13T11:46:45.519713+0000 mon.smithi088 (mon.0) 71 : cluster [INF] Active manager daemon smithi088.xwlxdh restarted 2024-01-13T11:46:45.762 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:45 smithi088 bash[14069]: cluster 2024-01-13T11:46:45.520443+0000 mon.smithi088 (mon.0) 72 : cluster [INF] Activating manager daemon smithi088.xwlxdh 2024-01-13T11:46:45.762 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:45 smithi088 bash[14069]: cluster 2024-01-13T11:46:45.527433+0000 mon.smithi088 (mon.0) 73 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-01-13T11:46:45.762 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:45 smithi088 bash[14069]: cluster 2024-01-13T11:46:45.527753+0000 mon.smithi088 (mon.0) 74 : cluster [DBG] mgrmap e10: smithi088.xwlxdh(active, starting, since 0.00761243s) 2024-01-13T11:46:45.762 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:45 smithi088 bash[14069]: audit 2024-01-13T11:46:45.528822+0000 mon.smithi088 (mon.0) 75 : audit [DBG] from='mgr.14156 172.21.15.88:0/2957041966' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mon metadata", "id": "smithi088"}]: dispatch 2024-01-13T11:46:45.762 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:45 smithi088 bash[14069]: audit 2024-01-13T11:46:45.530318+0000 mon.smithi088 (mon.0) 76 : audit [DBG] from='mgr.14156 172.21.15.88:0/2957041966' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mgr metadata", "who": "smithi088.xwlxdh", "id": "smithi088.xwlxdh"}]: dispatch 2024-01-13T11:46:45.762 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:45 smithi088 bash[14069]: audit 2024-01-13T11:46:45.531353+0000 mon.smithi088 (mon.0) 77 : audit [DBG] from='mgr.14156 172.21.15.88:0/2957041966' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-13T11:46:45.762 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:45 smithi088 bash[14069]: audit 2024-01-13T11:46:45.531549+0000 mon.smithi088 (mon.0) 78 : audit [DBG] from='mgr.14156 172.21.15.88:0/2957041966' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-13T11:46:45.762 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:45 smithi088 bash[14069]: audit 2024-01-13T11:46:45.531724+0000 mon.smithi088 (mon.0) 79 : audit [DBG] from='mgr.14156 172.21.15.88:0/2957041966' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-13T11:46:45.762 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:45 smithi088 bash[14069]: cluster 2024-01-13T11:46:45.535146+0000 mon.smithi088 (mon.0) 80 : cluster [INF] Manager daemon smithi088.xwlxdh is now available 2024-01-13T11:46:46.538 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: { 2024-01-13T11:46:46.538 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-01-13T11:46:46.538 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: "initialized": true 2024-01-13T11:46:46.538 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: } 2024-01-13T11:46:46.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:46 smithi088 bash[14069]: audit 2024-01-13T11:46:45.661872+0000 mon.smithi088 (mon.0) 81 : audit [DBG] from='mgr.14156 172.21.15.88:0/2957041966' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:46:46.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:46 smithi088 bash[14069]: audit 2024-01-13T11:46:45.665433+0000 mon.smithi088 (mon.0) 82 : audit [DBG] from='mgr.14156 172.21.15.88:0/2957041966' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:46:46.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:46 smithi088 bash[14069]: audit 2024-01-13T11:46:45.703315+0000 mon.smithi088 (mon.0) 83 : audit [INF] from='mgr.14156 172.21.15.88:0/2957041966' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi088.xwlxdh/mirror_snapshot_schedule"}]: dispatch 2024-01-13T11:46:46.895 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:46 smithi088 bash[14069]: audit 2024-01-13T11:46:45.708227+0000 mon.smithi088 (mon.0) 84 : audit [INF] from='mgr.14156 172.21.15.88:0/2957041966' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi088.xwlxdh/trash_purge_schedule"}]: dispatch 2024-01-13T11:46:46.895 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:46 smithi088 bash[14069]: cluster 2024-01-13T11:46:46.530997+0000 mon.smithi088 (mon.0) 85 : cluster [DBG] mgrmap e11: smithi088.xwlxdh(active, since 1.01086s) 2024-01-13T11:46:47.300 INFO:teuthology.orchestra.run.smithi088.stderr:mgr epoch 9 is available 2024-01-13T11:46:47.300 INFO:teuthology.orchestra.run.smithi088.stderr:Generating a dashboard self-signed certificate... 2024-01-13T11:46:47.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:47 smithi088 bash[14069]: audit 2024-01-13T11:46:46.530127+0000 mgr.smithi088.xwlxdh (mgr.14156) 1 : audit [DBG] from='client.14160 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-13T11:46:47.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:47 smithi088 bash[14069]: audit 2024-01-13T11:46:46.544898+0000 mgr.smithi088.xwlxdh (mgr.14156) 2 : audit [DBG] from='client.14160 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-13T11:46:48.649 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: Self-signed certificate created 2024-01-13T11:46:48.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:48 smithi088 bash[14069]: cluster 2024-01-13T11:46:47.584480+0000 mon.smithi088 (mon.0) 86 : cluster [DBG] mgrmap e12: smithi088.xwlxdh(active, since 2s) 2024-01-13T11:46:49.307 INFO:teuthology.orchestra.run.smithi088.stderr:Creating initial admin user... 2024-01-13T11:46:50.065 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:49 smithi088 bash[14069]: audit 2024-01-13T11:46:48.589640+0000 mgr.smithi088.xwlxdh (mgr.14156) 3 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T11:46:50.065 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:49 smithi088 bash[14069]: audit 2024-01-13T11:46:48.654020+0000 mon.smithi088 (mon.0) 87 : audit [INF] from='mgr.14156 172.21.15.88:0/2957041966' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:46:50.065 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:49 smithi088 bash[14069]: audit 2024-01-13T11:46:48.656655+0000 mon.smithi088 (mon.0) 88 : audit [INF] from='mgr.14156 172.21.15.88:0/2957041966' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:46:50.884 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$KAxrXWruSxbyjAChhNum4OEyVxmRo6G012lyLekfMQZW.1zi/b6t.", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1705146410, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-01-13T11:46:51.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:50 smithi088 bash[14069]: audit 2024-01-13T11:46:50.649288+0000 mgr.smithi088.xwlxdh (mgr.14156) 4 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T11:46:51.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:50 smithi088 bash[14069]: audit 2024-01-13T11:46:50.890855+0000 mon.smithi088 (mon.0) 89 : audit [INF] from='mgr.14156 172.21.15.88:0/2957041966' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:46:51.555 INFO:teuthology.orchestra.run.smithi088.stderr:Fetching dashboard port number... 2024-01-13T11:46:52.893 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: 8443 2024-01-13T11:46:53.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:52 smithi088 bash[14069]: cluster 2024-01-13T11:46:51.911849+0000 mon.smithi088 (mon.0) 90 : cluster [DBG] mgrmap e13: smithi088.xwlxdh(active, since 6s) 2024-01-13T11:46:53.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:52 smithi088 bash[14069]: audit 2024-01-13T11:46:52.603759+0000 mon.smithi088 (mon.0) 91 : audit [INF] from='mgr.14156 172.21.15.88:0/2957041966' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:46:53.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:52 smithi088 bash[14069]: audit 2024-01-13T11:46:52.901285+0000 mon.smithi088 (mon.0) 92 : audit [DBG] from='client.? 172.21.15.88:0/1297769767' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-01-13T11:46:53.589 INFO:teuthology.orchestra.run.smithi088.stderr:firewalld does not appear to be present 2024-01-13T11:46:53.589 INFO:teuthology.orchestra.run.smithi088.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-01-13T11:46:53.592 INFO:teuthology.orchestra.run.smithi088.stderr:Ceph Dashboard is now available at: 2024-01-13T11:46:53.593 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-01-13T11:46:53.593 INFO:teuthology.orchestra.run.smithi088.stderr: URL: https://smithi088.front.sepia.ceph.com:8443/ 2024-01-13T11:46:53.593 INFO:teuthology.orchestra.run.smithi088.stderr: User: admin 2024-01-13T11:46:53.593 INFO:teuthology.orchestra.run.smithi088.stderr: Password: l8nkky45qg 2024-01-13T11:46:53.593 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-01-13T11:46:53.593 INFO:teuthology.orchestra.run.smithi088.stderr:Enabling autotune for osd_memory_target 2024-01-13T11:46:57.129 INFO:teuthology.orchestra.run.smithi088.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-01-13T11:46:58.171 INFO:teuthology.orchestra.run.smithi088.stderr:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-01-13T11:46:58.171 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-01-13T11:46:58.171 INFO:teuthology.orchestra.run.smithi088.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 236db248-b209-11ee-95ac-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-01-13T11:46:58.171 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-01-13T11:46:58.171 INFO:teuthology.orchestra.run.smithi088.stderr:Or, if you are only running a single cluster on this host: 2024-01-13T11:46:58.171 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-01-13T11:46:58.171 INFO:teuthology.orchestra.run.smithi088.stderr: sudo /home/ubuntu/cephtest/cephadm shell 2024-01-13T11:46:58.172 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-01-13T11:46:58.172 INFO:teuthology.orchestra.run.smithi088.stderr:Please consider enabling telemetry to help improve Ceph: 2024-01-13T11:46:58.172 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-01-13T11:46:58.172 INFO:teuthology.orchestra.run.smithi088.stderr: ceph telemetry on 2024-01-13T11:46:58.172 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-01-13T11:46:58.172 INFO:teuthology.orchestra.run.smithi088.stderr:For more information see: 2024-01-13T11:46:58.172 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-01-13T11:46:58.172 INFO:teuthology.orchestra.run.smithi088.stderr: https://docs.ceph.com/en/pacific/mgr/telemetry/ 2024-01-13T11:46:58.172 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-01-13T11:46:58.172 INFO:teuthology.orchestra.run.smithi088.stderr:Bootstrap complete. 2024-01-13T11:46:58.203 INFO:tasks.cephadm:Fetching config... 2024-01-13T11:46:58.203 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-01-13T11:46:58.203 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-01-13T11:46:58.209 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-01-13T11:46:58.209 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-01-13T11:46:58.209 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-01-13T11:46:58.258 INFO:tasks.cephadm:Fetching mon keyring... 2024-01-13T11:46:58.259 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-01-13T11:46:58.259 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi088/keyring of=/dev/stdout 2024-01-13T11:46:58.307 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:46:58 smithi088 bash[14069]: audit 2024-01-13T11:46:57.133312+0000 mon.smithi088 (mon.0) 93 : audit [INF] from='client.? 172.21.15.88:0/99347503' entity='client.admin' 2024-01-13T11:46:58.311 INFO:tasks.cephadm:Fetching pub ssh key... 2024-01-13T11:46:58.311 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-01-13T11:46:58.311 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-01-13T11:46:58.356 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-01-13T11:46:58.357 DEBUG:teuthology.orchestra.run.smithi088:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDrdl3yroTiZdGfEf9MssfN62V1BN1jftfPO7qihpK7W3au9489FxkX8lvDgihOw/QaEx4j6jNsEFL494tw/kp1oMCEUUMW26j9fe5Peb1R9HsntdJvxAnTRys6ipMJOhs4fO0PdAC5kxVSOpGicwsVkKrtwNpxmToHB38HYHniG2OTvwJ0U7ei1SF0k7vSRxHJVNw4bL7mMG6Cy2cvFDbUNRQkWD5t6DDNhlbjbFM9cFw8tH8WA2nyVl30yFZmtBhoojBGCA8no374MTeDeI5yjwh+qpmj7bo59zKExmPxS71mf+/VshPefkdUQar/aCMSB/W2I/IsjAP1SKRFpbmJhe2jVR3714cimZ8LF89RRFiRbp1HAQG+vBmH08X6mc3sgL8zGSy3yj6tw2IpIKK8bcj/Zerg2hupKWQXs02pWWJX7kluYTHuDgTmqKema49Rv4YDDeOlfb55X78Zh2Inn2yU6Jb+OflD3enSangD+hHcUISjaug14NiQ3lkszdE= ceph-236db248-b209-11ee-95ac-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-13T11:46:58.420 INFO:teuthology.orchestra.run.smithi088.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDrdl3yroTiZdGfEf9MssfN62V1BN1jftfPO7qihpK7W3au9489FxkX8lvDgihOw/QaEx4j6jNsEFL494tw/kp1oMCEUUMW26j9fe5Peb1R9HsntdJvxAnTRys6ipMJOhs4fO0PdAC5kxVSOpGicwsVkKrtwNpxmToHB38HYHniG2OTvwJ0U7ei1SF0k7vSRxHJVNw4bL7mMG6Cy2cvFDbUNRQkWD5t6DDNhlbjbFM9cFw8tH8WA2nyVl30yFZmtBhoojBGCA8no374MTeDeI5yjwh+qpmj7bo59zKExmPxS71mf+/VshPefkdUQar/aCMSB/W2I/IsjAP1SKRFpbmJhe2jVR3714cimZ8LF89RRFiRbp1HAQG+vBmH08X6mc3sgL8zGSy3yj6tw2IpIKK8bcj/Zerg2hupKWQXs02pWWJX7kluYTHuDgTmqKema49Rv4YDDeOlfb55X78Zh2Inn2yU6Jb+OflD3enSangD+hHcUISjaug14NiQ3lkszdE= ceph-236db248-b209-11ee-95ac-87774f69a715 2024-01-13T11:46:58.426 DEBUG:teuthology.orchestra.run.smithi138:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDrdl3yroTiZdGfEf9MssfN62V1BN1jftfPO7qihpK7W3au9489FxkX8lvDgihOw/QaEx4j6jNsEFL494tw/kp1oMCEUUMW26j9fe5Peb1R9HsntdJvxAnTRys6ipMJOhs4fO0PdAC5kxVSOpGicwsVkKrtwNpxmToHB38HYHniG2OTvwJ0U7ei1SF0k7vSRxHJVNw4bL7mMG6Cy2cvFDbUNRQkWD5t6DDNhlbjbFM9cFw8tH8WA2nyVl30yFZmtBhoojBGCA8no374MTeDeI5yjwh+qpmj7bo59zKExmPxS71mf+/VshPefkdUQar/aCMSB/W2I/IsjAP1SKRFpbmJhe2jVR3714cimZ8LF89RRFiRbp1HAQG+vBmH08X6mc3sgL8zGSy3yj6tw2IpIKK8bcj/Zerg2hupKWQXs02pWWJX7kluYTHuDgTmqKema49Rv4YDDeOlfb55X78Zh2Inn2yU6Jb+OflD3enSangD+hHcUISjaug14NiQ3lkszdE= ceph-236db248-b209-11ee-95ac-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-13T11:46:58.450 INFO:teuthology.orchestra.run.smithi138.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDrdl3yroTiZdGfEf9MssfN62V1BN1jftfPO7qihpK7W3au9489FxkX8lvDgihOw/QaEx4j6jNsEFL494tw/kp1oMCEUUMW26j9fe5Peb1R9HsntdJvxAnTRys6ipMJOhs4fO0PdAC5kxVSOpGicwsVkKrtwNpxmToHB38HYHniG2OTvwJ0U7ei1SF0k7vSRxHJVNw4bL7mMG6Cy2cvFDbUNRQkWD5t6DDNhlbjbFM9cFw8tH8WA2nyVl30yFZmtBhoojBGCA8no374MTeDeI5yjwh+qpmj7bo59zKExmPxS71mf+/VshPefkdUQar/aCMSB/W2I/IsjAP1SKRFpbmJhe2jVR3714cimZ8LF89RRFiRbp1HAQG+vBmH08X6mc3sgL8zGSy3yj6tw2IpIKK8bcj/Zerg2hupKWQXs02pWWJX7kluYTHuDgTmqKema49Rv4YDDeOlfb55X78Zh2Inn2yU6Jb+OflD3enSangD+hHcUISjaug14NiQ3lkszdE= ceph-236db248-b209-11ee-95ac-87774f69a715 2024-01-13T11:46:58.462 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-01-13T11:47:03.483 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-01-13T11:47:03.483 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-01-13T11:47:03.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:03 smithi088 bash[14069]: audit 2024-01-13T11:47:02.395395+0000 mon.smithi088 (mon.0) 94 : audit [INF] from='client.? 172.21.15.88:0/2815192164' entity='client.admin' 2024-01-13T11:47:06.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:06 smithi088 bash[14069]: cluster 2024-01-13T11:47:05.533184+0000 mgr.smithi088.xwlxdh (mgr.14156) 5 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:47:06.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:06 smithi088 bash[14069]: audit 2024-01-13T11:47:06.461396+0000 mon.smithi088 (mon.0) 95 : audit [INF] from='mgr.14156 172.21.15.88:0/2957041966' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:47:07.739 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi138 2024-01-13T11:47:07.739 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-01-13T11:47:07.739 DEBUG:teuthology.orchestra.run.smithi138:> dd of=/etc/ceph/ceph.conf 2024-01-13T11:47:07.746 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-01-13T11:47:07.746 DEBUG:teuthology.orchestra.run.smithi138:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-13T11:47:07.796 INFO:tasks.cephadm:Adding host smithi138 to orchestrator... 2024-01-13T11:47:07.796 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph orch host add smithi138 2024-01-13T11:47:07.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:07 smithi088 bash[14069]: audit 2024-01-13T11:47:06.780355+0000 mon.smithi088 (mon.0) 96 : audit [INF] from='mgr.14156 172.21.15.88:0/2957041966' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config rm", "who": "osd/host:smithi088", "name": "osd_memory_target"}]: dispatch 2024-01-13T11:47:07.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:07 smithi088 bash[14069]: audit 2024-01-13T11:47:06.785541+0000 mon.smithi088 (mon.0) 97 : audit [INF] from='mgr.14156 172.21.15.88:0/2957041966' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:47:07.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:07 smithi088 bash[14069]: cephadm 2024-01-13T11:47:06.802029+0000 mgr.smithi088.xwlxdh (mgr.14156) 6 : cephadm [INF] Deploying daemon alertmanager.smithi088 on smithi088 2024-01-13T11:47:07.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:07 smithi088 bash[14069]: audit 2024-01-13T11:47:06.933522+0000 mon.smithi088 (mon.0) 98 : audit [INF] from='mgr.14156 172.21.15.88:0/2957041966' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:47:09.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:08 smithi088 bash[14069]: audit 2024-01-13T11:47:06.929959+0000 mgr.smithi088.xwlxdh (mgr.14156) 7 : audit [DBG] from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T11:47:09.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:08 smithi088 bash[14069]: cluster 2024-01-13T11:47:07.533577+0000 mgr.smithi088.xwlxdh (mgr.14156) 8 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:47:11.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:10 smithi088 bash[14069]: cluster 2024-01-13T11:47:09.533955+0000 mgr.smithi088.xwlxdh (mgr.14156) 9 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:47:13.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:12 smithi088 bash[14069]: cluster 2024-01-13T11:47:11.534320+0000 mgr.smithi088.xwlxdh (mgr.14156) 10 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:47:15.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:14 smithi088 bash[14069]: cluster 2024-01-13T11:47:13.534741+0000 mgr.smithi088.xwlxdh (mgr.14156) 11 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:47:16.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:16 smithi088 bash[14069]: cluster 2024-01-13T11:47:15.535234+0000 mgr.smithi088.xwlxdh (mgr.14156) 12 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:47:18.954 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:18 smithi088 bash[14069]: cluster 2024-01-13T11:47:17.535741+0000 mgr.smithi088.xwlxdh (mgr.14156) 13 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:47:18.954 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:18 smithi088 bash[14069]: audit 2024-01-13T11:47:18.721693+0000 mon.smithi088 (mon.0) 99 : audit [INF] from='mgr.14156 172.21.15.88:0/2957041966' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:47:18.954 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:18 smithi088 bash[14069]: audit 2024-01-13T11:47:18.725451+0000 mon.smithi088 (mon.0) 100 : audit [INF] from='mgr.14156 172.21.15.88:0/2957041966' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi088", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-13T11:47:18.955 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:18 smithi088 bash[14069]: audit 2024-01-13T11:47:18.730170+0000 mon.smithi088 (mon.0) 101 : audit [INF] from='mgr.14156 172.21.15.88:0/2957041966' entity='mgr.smithi088.xwlxdh' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi088", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-01-13T11:47:18.955 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:18 smithi088 bash[14069]: audit 2024-01-13T11:47:18.731357+0000 mon.smithi088 (mon.0) 102 : audit [DBG] from='mgr.14156 172.21.15.88:0/2957041966' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:47:20.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:19 smithi088 bash[14069]: cephadm 2024-01-13T11:47:18.732202+0000 mgr.smithi088.xwlxdh (mgr.14156) 14 : cephadm [INF] Deploying daemon crash.smithi088 on smithi088 2024-01-13T11:47:21.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:20 smithi088 bash[14069]: cluster 2024-01-13T11:47:19.536181+0000 mgr.smithi088.xwlxdh (mgr.14156) 15 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:47:21.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:20 smithi088 bash[14069]: audit 2024-01-13T11:47:20.712800+0000 mon.smithi088 (mon.0) 103 : audit [INF] from='mgr.14156 172.21.15.88:0/2957041966' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:47:23.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:22 smithi088 bash[14069]: cluster 2024-01-13T11:47:21.536488+0000 mgr.smithi088.xwlxdh (mgr.14156) 16 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:47:23.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:22 smithi088 bash[14069]: audit 2024-01-13T11:47:21.763083+0000 mon.smithi088 (mon.0) 104 : audit [INF] from='mgr.14156 172.21.15.88:0/2957041966' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:47:23.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:22 smithi088 bash[14069]: audit 2024-01-13T11:47:21.914212+0000 mon.smithi088 (mon.0) 105 : audit [INF] from='mgr.14156 172.21.15.88:0/2957041966' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:47:23.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:22 smithi088 bash[14069]: audit 2024-01-13T11:47:21.917653+0000 mon.smithi088 (mon.0) 106 : audit [INF] from='mgr.14156 172.21.15.88:0/2957041966' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:47:23.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:22 smithi088 bash[14069]: audit 2024-01-13T11:47:21.920927+0000 mon.smithi088 (mon.0) 107 : audit [INF] from='mgr.14156 172.21.15.88:0/2957041966' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-13T11:47:23.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:22 smithi088 bash[14069]: audit 2024-01-13T11:47:21.925151+0000 mon.smithi088 (mon.0) 108 : audit [INF] from='mgr.14156 172.21.15.88:0/2957041966' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:47:24.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:23 smithi088 bash[14069]: audit 2024-01-13T11:47:21.921463+0000 mgr.smithi088.xwlxdh (mgr.14156) 17 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-13T11:47:24.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:23 smithi088 bash[14069]: cephadm 2024-01-13T11:47:21.934932+0000 mgr.smithi088.xwlxdh (mgr.14156) 18 : cephadm [INF] Deploying daemon grafana.smithi088 on smithi088 2024-01-13T11:47:25.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:24 smithi088 bash[14069]: cluster 2024-01-13T11:47:23.536888+0000 mgr.smithi088.xwlxdh (mgr.14156) 19 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:47:27.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:26 smithi088 bash[14069]: cluster 2024-01-13T11:47:25.537442+0000 mgr.smithi088.xwlxdh (mgr.14156) 20 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:47:27.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:26 smithi088 bash[14069]: audit 2024-01-13T11:47:25.718355+0000 mon.smithi088 (mon.0) 109 : audit [INF] from='mgr.14156 172.21.15.88:0/2957041966' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:47:29.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:28 smithi088 bash[14069]: cluster 2024-01-13T11:47:27.537845+0000 mgr.smithi088.xwlxdh (mgr.14156) 21 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:47:31.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:30 smithi088 bash[14069]: cluster 2024-01-13T11:47:29.538239+0000 mgr.smithi088.xwlxdh (mgr.14156) 22 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:47:33.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:32 smithi088 bash[14069]: cluster 2024-01-13T11:47:31.538548+0000 mgr.smithi088.xwlxdh (mgr.14156) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:47:35.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:34 smithi088 bash[14069]: cluster 2024-01-13T11:47:33.538939+0000 mgr.smithi088.xwlxdh (mgr.14156) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:47:37.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:36 smithi088 bash[14069]: cluster 2024-01-13T11:47:35.539235+0000 mgr.smithi088.xwlxdh (mgr.14156) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:47:39.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:38 smithi088 bash[14069]: cluster 2024-01-13T11:47:37.539605+0000 mgr.smithi088.xwlxdh (mgr.14156) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:47:41.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:40 smithi088 bash[14069]: cluster 2024-01-13T11:47:39.539989+0000 mgr.smithi088.xwlxdh (mgr.14156) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:47:43.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:42 smithi088 bash[14069]: cluster 2024-01-13T11:47:41.540310+0000 mgr.smithi088.xwlxdh (mgr.14156) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:47:45.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:44 smithi088 bash[14069]: cluster 2024-01-13T11:47:43.540746+0000 mgr.smithi088.xwlxdh (mgr.14156) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:47:47.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:46 smithi088 bash[14069]: cluster 2024-01-13T11:47:45.541181+0000 mgr.smithi088.xwlxdh (mgr.14156) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:47:49.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:48 smithi088 bash[14069]: cluster 2024-01-13T11:47:47.541670+0000 mgr.smithi088.xwlxdh (mgr.14156) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:47:51.107 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:50 smithi088 bash[14069]: cluster 2024-01-13T11:47:49.542143+0000 mgr.smithi088.xwlxdh (mgr.14156) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:47:52.482 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:52 smithi088 bash[14069]: audit 2024-01-13T11:47:51.151473+0000 mon.smithi088 (mon.0) 110 : audit [INF] from='mgr.14156 172.21.15.88:0/2957041966' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:47:52.482 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:52 smithi088 bash[14069]: cephadm 2024-01-13T11:47:51.155765+0000 mgr.smithi088.xwlxdh (mgr.14156) 33 : cephadm [INF] Deploying daemon node-exporter.smithi088 on smithi088 2024-01-13T11:47:52.482 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:52 smithi088 bash[14069]: cluster 2024-01-13T11:47:51.542434+0000 mgr.smithi088.xwlxdh (mgr.14156) 34 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:47:52.482 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:52 smithi088 bash[14069]: audit 2024-01-13T11:47:51.622054+0000 mgr.smithi088.xwlxdh (mgr.14156) 35 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi138", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T11:47:52.482 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:52 smithi088 bash[14069]: audit 2024-01-13T11:47:51.860020+0000 mon.smithi088 (mon.0) 111 : audit [INF] from='mgr.14156 172.21.15.88:0/2957041966' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:47:52.482 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:52 smithi088 bash[14069]: audit 2024-01-13T11:47:51.863837+0000 mon.smithi088 (mon.0) 112 : audit [INF] from='mgr.14156 172.21.15.88:0/2957041966' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-01-13T11:47:52.488 INFO:teuthology.orchestra.run.smithi138.stdout:Added host 'smithi138' with addr '172.21.15.138' 2024-01-13T11:47:53.287 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph orch host ls --format=json 2024-01-13T11:47:53.601 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:53 smithi088 bash[14069]: audit 2024-01-13T11:47:52.495067+0000 mon.smithi088 (mon.0) 113 : audit [INF] from='mgr.14156 172.21.15.88:0/2957041966' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:47:53.602 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:53 smithi088 bash[14069]: audit 2024-01-13T11:47:52.866210+0000 mon.smithi088 (mon.0) 114 : audit [INF] from='mgr.14156 172.21.15.88:0/2957041966' entity='mgr.smithi088.xwlxdh' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-01-13T11:47:53.602 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:53 smithi088 bash[14069]: cluster 2024-01-13T11:47:52.866386+0000 mon.smithi088 (mon.0) 115 : cluster [DBG] mgrmap e14: smithi088.xwlxdh(active, since 67s) 2024-01-13T11:47:59.611 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:59 smithi088 bash[14069]: cluster 2024-01-13T11:47:59.527228+0000 mon.smithi088 (mon.0) 116 : cluster [INF] Active manager daemon smithi088.xwlxdh restarted 2024-01-13T11:47:59.611 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:59 smithi088 bash[14069]: cluster 2024-01-13T11:47:59.527948+0000 mon.smithi088 (mon.0) 117 : cluster [INF] Activating manager daemon smithi088.xwlxdh 2024-01-13T11:47:59.611 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:59 smithi088 bash[14069]: cluster 2024-01-13T11:47:59.534295+0000 mon.smithi088 (mon.0) 118 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-01-13T11:47:59.611 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:59 smithi088 bash[14069]: cluster 2024-01-13T11:47:59.534646+0000 mon.smithi088 (mon.0) 119 : cluster [DBG] mgrmap e15: smithi088.xwlxdh(active, starting, since 0.00698435s) 2024-01-13T11:47:59.611 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:59 smithi088 bash[14069]: audit 2024-01-13T11:47:59.535740+0000 mon.smithi088 (mon.0) 120 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mon metadata", "id": "smithi088"}]: dispatch 2024-01-13T11:47:59.611 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:59 smithi088 bash[14069]: audit 2024-01-13T11:47:59.536653+0000 mon.smithi088 (mon.0) 121 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mgr metadata", "who": "smithi088.xwlxdh", "id": "smithi088.xwlxdh"}]: dispatch 2024-01-13T11:47:59.611 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:59 smithi088 bash[14069]: audit 2024-01-13T11:47:59.537475+0000 mon.smithi088 (mon.0) 122 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-13T11:47:59.611 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:59 smithi088 bash[14069]: audit 2024-01-13T11:47:59.537950+0000 mon.smithi088 (mon.0) 123 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-13T11:47:59.611 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:59 smithi088 bash[14069]: audit 2024-01-13T11:47:59.538355+0000 mon.smithi088 (mon.0) 124 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-13T11:47:59.611 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:47:59 smithi088 bash[14069]: cluster 2024-01-13T11:47:59.541042+0000 mon.smithi088 (mon.0) 125 : cluster [INF] Manager daemon smithi088.xwlxdh is now available 2024-01-13T11:48:00.531 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-01-13T11:48:00.532 INFO:teuthology.orchestra.run.smithi138.stdout:[{"addr": "172.21.15.88", "hostname": "smithi088", "labels": [], "status": ""}, {"addr": "172.21.15.138", "hostname": "smithi138", "labels": [], "status": ""}] 2024-01-13T11:48:00.643 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:00 smithi088 bash[14069]: audit 2024-01-13T11:47:59.677773+0000 mon.smithi088 (mon.0) 126 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:48:00.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:00 smithi088 bash[14069]: audit 2024-01-13T11:47:59.682223+0000 mon.smithi088 (mon.0) 127 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:48:00.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:00 smithi088 bash[14069]: audit 2024-01-13T11:47:59.708372+0000 mon.smithi088 (mon.0) 128 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:00.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:00 smithi088 bash[14069]: audit 2024-01-13T11:47:59.739593+0000 mon.smithi088 (mon.0) 129 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi088.xwlxdh/mirror_snapshot_schedule"}]: dispatch 2024-01-13T11:48:00.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:00 smithi088 bash[14069]: audit 2024-01-13T11:47:59.748348+0000 mon.smithi088 (mon.0) 130 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi088.xwlxdh/trash_purge_schedule"}]: dispatch 2024-01-13T11:48:00.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:00 smithi088 bash[14069]: audit 2024-01-13T11:48:00.271013+0000 mon.smithi088 (mon.0) 131 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:00.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:00 smithi088 bash[14069]: cluster 2024-01-13T11:48:00.538311+0000 mon.smithi088 (mon.0) 132 : cluster [DBG] mgrmap e16: smithi088.xwlxdh(active, since 1.01066s) 2024-01-13T11:48:01.314 INFO:tasks.cephadm:Setting crush tunables to default 2024-01-13T11:48:01.314 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd crush tunables default 2024-01-13T11:48:01.715 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:01 smithi088 bash[14069]: audit 2024-01-13T11:48:00.538642+0000 mgr.smithi088.xwlxdh (mgr.14182) 1 : audit [DBG] from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-13T11:48:02.893 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:02 smithi088 bash[14069]: cluster 2024-01-13T11:48:01.591121+0000 mon.smithi088 (mon.0) 133 : cluster [DBG] mgrmap e17: smithi088.xwlxdh(active, since 2s) 2024-01-13T11:48:02.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:02 smithi088 bash[14069]: audit 2024-01-13T11:48:02.301738+0000 mon.smithi088 (mon.0) 134 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:06.811 INFO:teuthology.orchestra.run.smithi088.stderr:adjusted tunables profile to default 2024-01-13T11:48:06.970 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:06 smithi088 bash[14069]: audit 2024-01-13T11:48:05.805692+0000 mon.smithi088 (mon.0) 135 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:06.970 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:06 smithi088 bash[14069]: audit 2024-01-13T11:48:06.071907+0000 mon.smithi088 (mon.0) 136 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config rm", "who": "osd/host:smithi088", "name": "osd_memory_target"}]: dispatch 2024-01-13T11:48:06.970 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:06 smithi088 bash[14069]: audit 2024-01-13T11:48:06.227303+0000 mon.smithi088 (mon.0) 137 : audit [INF] from='client.? 172.21.15.88:0/2551714215' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-01-13T11:48:07.940 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:07 smithi088 bash[14069]: audit 2024-01-13T11:48:06.816384+0000 mon.smithi088 (mon.0) 138 : audit [INF] from='client.? 172.21.15.88:0/2551714215' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-01-13T11:48:07.941 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:07 smithi088 bash[14069]: cluster 2024-01-13T11:48:06.816447+0000 mon.smithi088 (mon.0) 139 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-01-13T11:48:08.014 INFO:tasks.cephadm:Adding mon.smithi088 on smithi088 2024-01-13T11:48:08.014 INFO:tasks.cephadm:Adding mon.smithi138 on smithi138 2024-01-13T11:48:08.015 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph orch apply mon '2;smithi088:172.21.15.88=smithi088;smithi138:172.21.15.138=smithi138' 2024-01-13T11:48:08.977 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:08 smithi088 bash[14069]: audit 2024-01-13T11:48:07.916381+0000 mon.smithi088 (mon.0) 140 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:08.977 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:08 smithi088 bash[14069]: audit 2024-01-13T11:48:08.237179+0000 mon.smithi088 (mon.0) 141 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:08.977 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:08 smithi088 bash[14069]: audit 2024-01-13T11:48:08.238459+0000 mon.smithi088 (mon.0) 142 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config rm", "who": "osd/host:smithi138", "name": "osd_memory_target"}]: dispatch 2024-01-13T11:48:08.977 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:08 smithi088 bash[14069]: audit 2024-01-13T11:48:08.239913+0000 mon.smithi088 (mon.0) 143 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:48:08.977 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:08 smithi088 bash[14069]: audit 2024-01-13T11:48:08.241234+0000 mon.smithi088 (mon.0) 144 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T11:48:08.978 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:08 smithi088 bash[14069]: audit 2024-01-13T11:48:08.330748+0000 mon.smithi088 (mon.0) 145 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:08.978 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:08 smithi088 bash[14069]: audit 2024-01-13T11:48:08.563774+0000 mon.smithi088 (mon.0) 146 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:08.978 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:08 smithi088 bash[14069]: audit 2024-01-13T11:48:08.570391+0000 mon.smithi088 (mon.0) 147 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:08.978 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:08 smithi088 bash[14069]: audit 2024-01-13T11:48:08.572545+0000 mon.smithi088 (mon.0) 148 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi138", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-13T11:48:08.978 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:08 smithi088 bash[14069]: audit 2024-01-13T11:48:08.577163+0000 mon.smithi088 (mon.0) 149 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi138", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-01-13T11:48:08.978 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:08 smithi088 bash[14069]: audit 2024-01-13T11:48:08.578165+0000 mon.smithi088 (mon.0) 150 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:48:09.987 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:09 smithi088 bash[14069]: cephadm 2024-01-13T11:48:08.243657+0000 mgr.smithi088.xwlxdh (mgr.14182) 2 : cephadm [INF] Updating smithi088:/etc/ceph/ceph.conf 2024-01-13T11:48:09.988 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:09 smithi088 bash[14069]: cephadm 2024-01-13T11:48:08.244250+0000 mgr.smithi088.xwlxdh (mgr.14182) 3 : cephadm [INF] Updating smithi138:/etc/ceph/ceph.conf 2024-01-13T11:48:09.988 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:09 smithi088 bash[14069]: cephadm 2024-01-13T11:48:08.270243+0000 mgr.smithi088.xwlxdh (mgr.14182) 4 : cephadm [INF] Updating smithi088:/etc/ceph/ceph.client.admin.keyring 2024-01-13T11:48:09.988 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:09 smithi088 bash[14069]: cephadm 2024-01-13T11:48:08.361967+0000 mgr.smithi088.xwlxdh (mgr.14182) 5 : cephadm [INF] Updating smithi138:/etc/ceph/ceph.client.admin.keyring 2024-01-13T11:48:09.988 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:09 smithi088 bash[14069]: cephadm 2024-01-13T11:48:08.578979+0000 mgr.smithi088.xwlxdh (mgr.14182) 6 : cephadm [INF] Deploying daemon crash.smithi138 on smithi138 2024-01-13T11:48:12.054 INFO:teuthology.orchestra.run.smithi138.stdout:Scheduled mon update... 2024-01-13T11:48:13.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:13 smithi088 bash[14069]: audit 2024-01-13T11:48:12.052469+0000 mgr.smithi088.xwlxdh (mgr.14182) 7 : audit [DBG] from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi088:172.21.15.88=smithi088;smithi138:172.21.15.138=smithi138", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T11:48:13.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:13 smithi088 bash[14069]: cephadm 2024-01-13T11:48:12.058130+0000 mgr.smithi088.xwlxdh (mgr.14182) 8 : cephadm [INF] Saving service mon spec with placement smithi088:172.21.15.88=smithi088;smithi138:172.21.15.138=smithi138;count:2 2024-01-13T11:48:13.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:13 smithi088 bash[14069]: audit 2024-01-13T11:48:12.061949+0000 mon.smithi088 (mon.0) 151 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:13.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:13 smithi088 bash[14069]: audit 2024-01-13T11:48:12.463040+0000 mon.smithi088 (mon.0) 152 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:13.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:13 smithi088 bash[14069]: audit 2024-01-13T11:48:12.467093+0000 mon.smithi088 (mon.0) 153 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi138.divzmq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-13T11:48:13.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:13 smithi088 bash[14069]: audit 2024-01-13T11:48:12.472354+0000 mon.smithi088 (mon.0) 154 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi138.divzmq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-01-13T11:48:13.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:13 smithi088 bash[14069]: audit 2024-01-13T11:48:12.473688+0000 mon.smithi088 (mon.0) 155 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-13T11:48:13.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:13 smithi088 bash[14069]: audit 2024-01-13T11:48:12.475033+0000 mon.smithi088 (mon.0) 156 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:48:13.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:13 smithi088 bash[14069]: cephadm 2024-01-13T11:48:12.476091+0000 mgr.smithi088.xwlxdh (mgr.14182) 9 : cephadm [INF] Deploying daemon mgr.smithi138.divzmq on smithi138 2024-01-13T11:48:13.479 DEBUG:teuthology.orchestra.run.smithi138:mon.smithi138> sudo journalctl -f -n 0 -u ceph-236db248-b209-11ee-95ac-87774f69a715@mon.smithi138.service 2024-01-13T11:48:13.481 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:48:13.481 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:48:14.714 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2024-01-13T11:48:16.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:15 smithi088 bash[14069]: audit 2024-01-13T11:48:14.712137+0000 mon.smithi088 (mon.0) 157 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:17.893 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:17 smithi088 bash[14069]: audit 2024-01-13T11:48:16.594683+0000 mon.smithi088 (mon.0) 158 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:17.893 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:17 smithi088 bash[14069]: audit 2024-01-13T11:48:16.598311+0000 mon.smithi088 (mon.0) 159 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-13T11:48:17.893 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:17 smithi088 bash[14069]: audit 2024-01-13T11:48:16.599721+0000 mon.smithi088 (mon.0) 160 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-13T11:48:17.893 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:17 smithi088 bash[14069]: audit 2024-01-13T11:48:16.601046+0000 mon.smithi088 (mon.0) 161 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:48:17.893 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:17 smithi088 bash[14069]: cephadm 2024-01-13T11:48:16.602209+0000 mgr.smithi088.xwlxdh (mgr.14182) 10 : cephadm [INF] Deploying daemon mon.smithi138 on smithi138 2024-01-13T11:48:18.571 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-01-13T11:48:18.571 INFO:teuthology.orchestra.run.smithi138.stdout:{"epoch":1,"fsid":"236db248-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:45:31.831492Z","created":"2024-01-13T11:45:31.831492Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi088","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T11:48:18.573 INFO:teuthology.orchestra.run.smithi138.stderr:dumped monmap epoch 1 2024-01-13T11:48:19.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:18 smithi088 bash[14069]: audit 2024-01-13T11:48:18.579872+0000 mon.smithi088 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.138:0/441400201' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:48:20.877 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T11:48:20.878 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T11:48:21.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:20 smithi088 bash[14069]: cluster 2024-01-13T11:48:19.539145+0000 mgr.smithi088.xwlxdh (mgr.14182) 11 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:21.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:20 smithi088 bash[14069]: audit 2024-01-13T11:48:19.718378+0000 mon.smithi088 (mon.0) 163 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:21.557 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:21 smithi138 systemd[1]: Started Ceph mon.smithi138 for 236db248-b209-11ee-95ac-87774f69a715. 2024-01-13T11:48:22.893 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:22 smithi088 bash[14069]: cluster 2024-01-13T11:48:21.539599+0000 mgr.smithi088.xwlxdh (mgr.14182) 12 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:22.893 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:22 smithi088 bash[14069]: audit 2024-01-13T11:48:21.549987+0000 mon.smithi088 (mon.0) 164 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:22.893 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:22 smithi088 bash[14069]: cephadm 2024-01-13T11:48:21.553129+0000 mgr.smithi088.xwlxdh (mgr.14182) 13 : cephadm [INF] Deploying daemon node-exporter.smithi138 on smithi138 2024-01-13T11:48:22.893 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:22 smithi088 bash[14069]: audit 2024-01-13T11:48:22.216394+0000 mon.smithi088 (mon.0) 165 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:23.877 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.714+0000 7f3d1234c880 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-13T11:48:23.877 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.714+0000 7f3d1234c880 0 ceph version 16.2.14-456-g445562ab (445562ab4bc3ddfb386936119050695810860bcb) pacific (stable), process ceph-mon, pid 7 2024-01-13T11:48:23.877 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.714+0000 7f3d1234c880 0 pidfile_write: ignore empty --pid-file 2024-01-13T11:48:23.878 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 0 load: jerasure load: lrc load: isa 2024-01-13T11:48:23.878 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: RocksDB version: 6.8.1 2024-01-13T11:48:23.878 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-13T11:48:23.878 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Compile date Jan 12 2024 2024-01-13T11:48:23.878 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: DB SUMMARY 2024-01-13T11:48:23.878 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: CURRENT file: CURRENT 2024-01-13T11:48:23.878 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: IDENTITY file: IDENTITY 2024-01-13T11:48:23.878 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: MANIFEST file: MANIFEST-000001 size: 13 Bytes 2024-01-13T11:48:23.878 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi138/store.db dir, Total Num: 0, files: 2024-01-13T11:48:23.879 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi138/store.db: 000003.log size: 511 ; 2024-01-13T11:48:23.879 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.error_if_exists: 0 2024-01-13T11:48:23.879 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.create_if_missing: 0 2024-01-13T11:48:23.879 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.paranoid_checks: 1 2024-01-13T11:48:23.879 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.env: 0x55db771c3080 2024-01-13T11:48:23.879 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.fs: Posix File System 2024-01-13T11:48:23.879 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.info_log: 0x55db78dc0400 2024-01-13T11:48:23.879 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-13T11:48:23.879 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.statistics: (nil) 2024-01-13T11:48:23.879 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.use_fsync: 0 2024-01-13T11:48:23.879 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.max_log_file_size: 0 2024-01-13T11:48:23.879 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-13T11:48:23.879 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-13T11:48:23.880 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-13T11:48:23.880 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-13T11:48:23.880 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.allow_fallocate: 1 2024-01-13T11:48:23.880 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-13T11:48:23.880 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-13T11:48:23.880 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.use_direct_reads: 0 2024-01-13T11:48:23.880 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-13T11:48:23.880 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.create_missing_column_families: 0 2024-01-13T11:48:23.880 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.db_log_dir: 2024-01-13T11:48:23.880 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi138/store.db 2024-01-13T11:48:23.880 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-13T11:48:23.881 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.max_subcompactions: 1 2024-01-13T11:48:23.881 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.max_background_flushes: -1 2024-01-13T11:48:23.881 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-13T11:48:23.881 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-13T11:48:23.881 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-13T11:48:23.881 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-13T11:48:23.881 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-13T11:48:23.881 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.advise_random_on_open: 1 2024-01-13T11:48:23.881 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-13T11:48:23.881 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.write_buffer_manager: 0x55db79a0e480 2024-01-13T11:48:23.881 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-13T11:48:23.881 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-13T11:48:23.881 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-13T11:48:23.882 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-13T11:48:23.882 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.rate_limiter: (nil) 2024-01-13T11:48:23.882 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-13T11:48:23.882 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-13T11:48:23.882 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-13T11:48:23.882 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-13T11:48:23.883 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.unordered_write: 0 2024-01-13T11:48:23.883 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-13T11:48:23.883 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-13T11:48:23.883 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-13T11:48:23.883 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-13T11:48:23.883 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.row_cache: None 2024-01-13T11:48:23.883 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.wal_filter: None 2024-01-13T11:48:23.884 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-13T11:48:23.884 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-13T11:48:23.884 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.preserve_deletes: 0 2024-01-13T11:48:23.884 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.two_write_queues: 0 2024-01-13T11:48:23.884 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.manual_wal_flush: 0 2024-01-13T11:48:23.884 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.atomic_flush: 0 2024-01-13T11:48:23.885 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-13T11:48:23.885 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-13T11:48:23.885 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-13T11:48:23.885 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.log_readahead_size: 0 2024-01-13T11:48:23.885 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-13T11:48:23.885 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.max_background_jobs: 2 2024-01-13T11:48:23.886 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.max_background_compactions: -1 2024-01-13T11:48:23.886 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-13T11:48:23.886 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-13T11:48:23.886 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-13T11:48:23.886 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.max_total_wal_size: 0 2024-01-13T11:48:23.886 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-13T11:48:23.886 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-13T11:48:23.887 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-13T11:48:23.887 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-13T11:48:23.887 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.max_open_files: -1 2024-01-13T11:48:23.887 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.bytes_per_sync: 0 2024-01-13T11:48:23.887 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-13T11:48:23.887 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-13T11:48:23.887 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-13T11:48:23.887 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Compression algorithms supported: 2024-01-13T11:48:23.887 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-13T11:48:23.887 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: kZSTD supported: 0 2024-01-13T11:48:23.888 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: kXpressCompression supported: 0 2024-01-13T11:48:23.888 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-13T11:48:23.888 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: kLZ4Compression supported: 1 2024-01-13T11:48:23.888 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: kBZip2Compression supported: 0 2024-01-13T11:48:23.888 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: kZlibCompression supported: 1 2024-01-13T11:48:23.888 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: kSnappyCompression supported: 1 2024-01-13T11:48:23.888 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-13T11:48:23.888 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi138/store.db/MANIFEST-000001 2024-01-13T11:48:23.888 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-13T11:48:23.888 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-13T11:48:23.889 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.merge_operator: 2024-01-13T11:48:23.889 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.compaction_filter: None 2024-01-13T11:48:23.889 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.compaction_filter_factory: None 2024-01-13T11:48:23.889 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-13T11:48:23.889 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-13T11:48:23.889 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55db78d63d28) 2024-01-13T11:48:23.889 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: cache_index_and_filter_blocks: 1 2024-01-13T11:48:23.889 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-13T11:48:23.889 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-13T11:48:23.889 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: pin_top_level_index_and_filter: 1 2024-01-13T11:48:23.890 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: index_type: 0 2024-01-13T11:48:23.890 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: data_block_index_type: 0 2024-01-13T11:48:23.890 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: index_shortening: 1 2024-01-13T11:48:23.890 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: data_block_hash_table_util_ratio: 0.750000 2024-01-13T11:48:23.890 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: hash_index_allow_collision: 1 2024-01-13T11:48:23.890 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: checksum: 1 2024-01-13T11:48:23.890 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: no_block_cache: 0 2024-01-13T11:48:23.890 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: block_cache: 0x55db78d9af10 2024-01-13T11:48:23.890 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: block_cache_name: BinnedLRUCache 2024-01-13T11:48:23.890 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: block_cache_options: 2024-01-13T11:48:23.891 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: capacity : 536870912 2024-01-13T11:48:23.891 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: num_shard_bits : 4 2024-01-13T11:48:23.891 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: strict_capacity_limit : 0 2024-01-13T11:48:23.891 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: high_pri_pool_ratio: 0.000 2024-01-13T11:48:23.891 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: block_cache_compressed: (nil) 2024-01-13T11:48:23.891 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: persistent_cache: (nil) 2024-01-13T11:48:23.891 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: block_size: 4096 2024-01-13T11:48:23.891 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: block_size_deviation: 10 2024-01-13T11:48:23.891 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: block_restart_interval: 16 2024-01-13T11:48:23.891 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: index_block_restart_interval: 1 2024-01-13T11:48:23.892 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: metadata_block_size: 4096 2024-01-13T11:48:23.892 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: partition_filters: 0 2024-01-13T11:48:23.892 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: use_delta_encoding: 1 2024-01-13T11:48:23.892 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-13T11:48:23.892 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: whole_key_filtering: 1 2024-01-13T11:48:23.892 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: verify_compression: 0 2024-01-13T11:48:23.892 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: read_amp_bytes_per_bit: 0 2024-01-13T11:48:23.892 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: format_version: 2 2024-01-13T11:48:23.892 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: enable_index_compression: 1 2024-01-13T11:48:23.892 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: block_align: 0 2024-01-13T11:48:23.893 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-13T11:48:23.893 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-13T11:48:23.893 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.compression: NoCompression 2024-01-13T11:48:23.893 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-13T11:48:23.893 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-13T11:48:23.893 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-13T11:48:23.893 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.num_levels: 7 2024-01-13T11:48:23.893 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-13T11:48:23.893 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-13T11:48:23.893 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-13T11:48:23.894 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-13T11:48:23.894 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-13T11:48:23.894 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-13T11:48:23.894 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-13T11:48:23.894 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-13T11:48:23.894 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-13T11:48:23.894 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-13T11:48:23.894 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.compression_opts.level: 32767 2024-01-13T11:48:23.894 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-13T11:48:23.894 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-13T11:48:23.894 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-13T11:48:23.895 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.compression_opts.enabled: false 2024-01-13T11:48:23.895 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-13T11:48:23.895 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-13T11:48:23.895 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-13T11:48:23.895 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-13T11:48:23.895 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-13T11:48:23.895 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-13T11:48:23.895 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-13T11:48:23.895 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-13T11:48:23.895 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-13T11:48:23.896 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-13T11:48:23.896 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-13T11:48:23.896 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-13T11:48:23.896 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-13T11:48:23.896 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-13T11:48:23.896 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-13T11:48:23.896 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-13T11:48:23.896 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-13T11:48:23.896 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.arena_block_size: 4194304 2024-01-13T11:48:23.897 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-13T11:48:23.897 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-13T11:48:23.897 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-13T11:48:23.897 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-13T11:48:23.897 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-13T11:48:23.897 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-13T11:48:23.897 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-13T11:48:23.897 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-13T11:48:23.897 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-13T11:48:23.897 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-13T11:48:23.898 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-13T11:48:23.898 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-13T11:48:23.898 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-13T11:48:23.898 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-13T11:48:23.898 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.table_properties_collectors: 2024-01-13T11:48:23.898 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.inplace_update_support: 0 2024-01-13T11:48:23.898 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-13T11:48:23.898 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-13T11:48:23.898 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-13T11:48:23.898 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-13T11:48:23.899 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.bloom_locality: 0 2024-01-13T11:48:23.899 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.max_successive_merges: 0 2024-01-13T11:48:23.899 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-13T11:48:23.899 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-13T11:48:23.899 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.force_consistency_checks: 0 2024-01-13T11:48:23.899 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-13T11:48:23.899 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.ttl: 2592000 2024-01-13T11:48:23.899 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-13T11:48:23.899 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi138/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-13T11:48:23.899 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-01-13T11:48:23.900 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705146503726459, "job": 1, "event": "recovery_started", "log_files": [3]} 2024-01-13T11:48:23.900 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #3 mode 2 2024-01-13T11:48:23.900 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-01-13T11:48:23.900 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705146503727002, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 4, "file_size": 1437, "table_properties": {"data_size": 523, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1705146503, "oldest_key_time": 3, "file_creation_time": 0}} 2024-01-13T11:48:23.900 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: [version_set.cc:3826] Creating manifest 5 2024-01-13T11:48:23.900 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.722+0000 7f3d1234c880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705146503727425, "job": 1, "event": "recovery_finished"} 2024-01-13T11:48:23.900 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.726+0000 7f3d1234c880 4 rocksdb: DB pointer 0x55db78e63800 2024-01-13T11:48:23.900 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.726+0000 7f3cf9140700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-13T11:48:23.900 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.726+0000 7f3cf9140700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-13T11:48:23.900 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: ** DB Stats ** 2024-01-13T11:48:23.901 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T11:48:23.901 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-13T11:48:23.901 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-13T11:48:23.901 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-13T11:48:23.901 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-13T11:48:23.901 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-13T11:48:23.901 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-13T11:48:23.901 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: ** Compaction Stats [default] ** 2024-01-13T11:48:23.901 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T11:48:23.902 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T11:48:23.902 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.7 0.00 0.00 1 0.000 0 0 2024-01-13T11:48:23.902 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.7 0.00 0.00 1 0.000 0 0 2024-01-13T11:48:23.902 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.7 0.00 0.00 1 0.000 0 0 2024-01-13T11:48:23.902 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: ** Compaction Stats [default] ** 2024-01-13T11:48:23.902 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T11:48:23.902 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T11:48:23.902 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.7 0.00 0.00 1 0.000 0 0 2024-01-13T11:48:23.902 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T11:48:23.902 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-13T11:48:23.903 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-13T11:48:23.903 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: AddFile(Total Files): cumulative 0, interval 0 2024-01-13T11:48:23.903 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-13T11:48:23.903 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: AddFile(Keys): cumulative 0, interval 0 2024-01-13T11:48:23.903 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: Cumulative compaction: 0.00 GB write, 0.44 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T11:48:23.903 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: Interval compaction: 0.00 GB write, 0.44 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T11:48:23.903 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-13T11:48:23.903 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: ** File Read Latency Histogram By Level [default] ** 2024-01-13T11:48:23.903 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: ** Compaction Stats [default] ** 2024-01-13T11:48:23.903 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T11:48:23.903 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T11:48:23.904 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.7 0.00 0.00 1 0.000 0 0 2024-01-13T11:48:23.904 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.7 0.00 0.00 1 0.000 0 0 2024-01-13T11:48:23.904 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-13T11:48:23.904 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: ** Compaction Stats [default] ** 2024-01-13T11:48:23.904 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T11:48:23.904 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T11:48:23.904 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.7 0.00 0.00 1 0.000 0 0 2024-01-13T11:48:23.904 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T11:48:23.904 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-13T11:48:23.905 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-13T11:48:23.905 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: AddFile(Total Files): cumulative 0, interval 0 2024-01-13T11:48:23.905 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-13T11:48:23.905 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: AddFile(Keys): cumulative 0, interval 0 2024-01-13T11:48:23.905 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: Cumulative compaction: 0.00 GB write, 0.43 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T11:48:23.905 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T11:48:23.905 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-13T11:48:23.905 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: ** File Read Latency Histogram By Level [default] ** 2024-01-13T11:48:23.905 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.726+0000 7f3d1234c880 0 mon.smithi138 does not exist in monmap, will attempt to join an existing cluster 2024-01-13T11:48:23.905 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.726+0000 7f3d1234c880 0 using public_addr v2:172.21.15.138:0/0 -> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-01-13T11:48:23.906 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.726+0000 7f3d1234c880 0 starting mon.smithi138 rank -1 at public addrs [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] at bind addrs [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi138 fsid 236db248-b209-11ee-95ac-87774f69a715 2024-01-13T11:48:23.906 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.726+0000 7f3d1234c880 1 mon.smithi138@-1(???) e0 preinit fsid 236db248-b209-11ee-95ac-87774f69a715 2024-01-13T11:48:23.906 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.758+0000 7f3cfb945700 0 mon.smithi138@-1(synchronizing).mds e1 new map 2024-01-13T11:48:23.906 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.758+0000 7f3cfb945700 0 mon.smithi138@-1(synchronizing).mds e1 print_map 2024-01-13T11:48:23.906 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: e1 2024-01-13T11:48:23.906 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: enable_multiple, ever_enabled_multiple: 1,1 2024-01-13T11:48:23.906 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-13T11:48:23.906 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: legacy client fscid: -1 2024-01-13T11:48:23.906 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: 2024-01-13T11:48:23.906 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: No filesystems configured 2024-01-13T11:48:23.906 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.758+0000 7f3cfb945700 1 mon.smithi138@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-01-13T11:48:23.907 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.758+0000 7f3cfb945700 1 mon.smithi138@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-01-13T11:48:23.907 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.758+0000 7f3cfb945700 1 mon.smithi138@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-01-13T11:48:23.907 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.758+0000 7f3cfb945700 1 mon.smithi138@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-01-13T11:48:23.907 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.758+0000 7f3cfb945700 1 mon.smithi138@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-01-13T11:48:23.907 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.758+0000 7f3cfb945700 1 mon.smithi138@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-01-13T11:48:23.907 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.758+0000 7f3cfb945700 1 mon.smithi138@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-01-13T11:48:23.907 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.758+0000 7f3cfb945700 0 mon.smithi138@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-01-13T11:48:23.907 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.758+0000 7f3cfb945700 0 mon.smithi138@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-01-13T11:48:23.907 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.758+0000 7f3cfb945700 0 mon.smithi138@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-01-13T11:48:23.908 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.758+0000 7f3cfb945700 0 mon.smithi138@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-01-13T11:48:23.908 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: cluster 2024-01-13T11:48:21.539599+0000 mgr.smithi088.xwlxdh (mgr.14182) 12 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:23.908 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: audit 2024-01-13T11:48:21.549987+0000 mon.smithi088 (mon.0) 164 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:23.908 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: cephadm 2024-01-13T11:48:21.553129+0000 mgr.smithi088.xwlxdh (mgr.14182) 13 : cephadm [INF] Deploying daemon node-exporter.smithi138 on smithi138 2024-01-13T11:48:23.908 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: audit 2024-01-13T11:48:22.216394+0000 mon.smithi088 (mon.0) 165 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:23.908 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:23 smithi138 bash[17620]: debug 2024-01-13T11:48:23.758+0000 7f3cfb945700 1 mon.smithi138@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-01-13T11:48:29.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:28 smithi088 bash[14069]: cluster 2024-01-13T11:48:23.540050+0000 mgr.smithi088.xwlxdh (mgr.14182) 15 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:29.190 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:28 smithi088 bash[14069]: audit 2024-01-13T11:48:23.767521+0000 mon.smithi088 (mon.0) 166 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mon metadata", "id": "smithi138"}]: dispatch 2024-01-13T11:48:29.191 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:28 smithi088 bash[14069]: audit 2024-01-13T11:48:23.771706+0000 mon.smithi088 (mon.0) 167 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mon metadata", "id": "smithi088"}]: dispatch 2024-01-13T11:48:29.191 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:28 smithi088 bash[14069]: cluster 2024-01-13T11:48:23.771957+0000 mon.smithi088 (mon.0) 168 : cluster [INF] mon.smithi088 calling monitor election 2024-01-13T11:48:29.191 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:28 smithi088 bash[14069]: audit 2024-01-13T11:48:23.790500+0000 mon.smithi088 (mon.0) 169 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mon metadata", "id": "smithi138"}]: dispatch 2024-01-13T11:48:29.191 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:28 smithi088 bash[14069]: audit 2024-01-13T11:48:24.767016+0000 mon.smithi088 (mon.0) 170 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mon metadata", "id": "smithi138"}]: dispatch 2024-01-13T11:48:29.191 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:28 smithi088 bash[14069]: audit 2024-01-13T11:48:25.017175+0000 mon.smithi088 (mon.0) 171 : audit [DBG] from='mgr.? 172.21.15.138:0/2380498778' entity='mgr.smithi138.divzmq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi138.divzmq/crt"}]: dispatch 2024-01-13T11:48:29.191 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:28 smithi088 bash[14069]: cluster 2024-01-13T11:48:25.540541+0000 mgr.smithi088.xwlxdh (mgr.14182) 16 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:29.191 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:28 smithi088 bash[14069]: audit 2024-01-13T11:48:25.767301+0000 mon.smithi088 (mon.0) 172 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mon metadata", "id": "smithi138"}]: dispatch 2024-01-13T11:48:29.191 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:28 smithi088 bash[14069]: cluster 2024-01-13T11:48:25.768960+0000 mon.smithi138 (mon.1) 1 : cluster [INF] mon.smithi138 calling monitor election 2024-01-13T11:48:29.191 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:28 smithi088 bash[14069]: audit 2024-01-13T11:48:26.767349+0000 mon.smithi088 (mon.0) 173 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mon metadata", "id": "smithi138"}]: dispatch 2024-01-13T11:48:29.191 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:28 smithi088 bash[14069]: cluster 2024-01-13T11:48:27.541045+0000 mgr.smithi088.xwlxdh (mgr.14182) 17 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:29.192 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:28 smithi088 bash[14069]: audit 2024-01-13T11:48:27.768064+0000 mon.smithi088 (mon.0) 174 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mon metadata", "id": "smithi138"}]: dispatch 2024-01-13T11:48:29.192 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:28 smithi088 bash[14069]: audit 2024-01-13T11:48:28.767768+0000 mon.smithi088 (mon.0) 175 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mon metadata", "id": "smithi138"}]: dispatch 2024-01-13T11:48:29.192 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:28 smithi088 bash[14069]: cluster 2024-01-13T11:48:28.810800+0000 mon.smithi088 (mon.0) 176 : cluster [INF] mon.smithi088 is new leader, mons smithi088,smithi138 in quorum (ranks 0,1) 2024-01-13T11:48:29.192 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:28 smithi088 bash[14069]: cluster 2024-01-13T11:48:28.816108+0000 mon.smithi088 (mon.0) 177 : cluster [DBG] monmap e2: 2 mons at {smithi088=[v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0],smithi138=[v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0]} removed_ranks: {} 2024-01-13T11:48:29.192 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:28 smithi088 bash[14069]: cluster 2024-01-13T11:48:28.834235+0000 mon.smithi088 (mon.0) 178 : cluster [DBG] fsmap 2024-01-13T11:48:29.192 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:28 smithi088 bash[14069]: cluster 2024-01-13T11:48:28.834335+0000 mon.smithi088 (mon.0) 179 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-01-13T11:48:29.192 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:28 smithi088 bash[14069]: cluster 2024-01-13T11:48:28.834755+0000 mon.smithi088 (mon.0) 180 : cluster [DBG] mgrmap e17: smithi088.xwlxdh(active, since 29s) 2024-01-13T11:48:29.192 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:28 smithi088 bash[14069]: cluster 2024-01-13T11:48:28.836847+0000 mon.smithi088 (mon.0) 181 : cluster [DBG] Standby manager daemon smithi138.divzmq started 2024-01-13T11:48:29.192 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:28 smithi088 bash[14069]: cluster 2024-01-13T11:48:28.843724+0000 mon.smithi088 (mon.0) 182 : cluster [INF] overall HEALTH_OK 2024-01-13T11:48:29.192 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:28 smithi088 bash[14069]: audit 2024-01-13T11:48:28.844446+0000 mon.smithi088 (mon.0) 183 : audit [DBG] from='mgr.? 172.21.15.138:0/2380498778' entity='mgr.smithi138.divzmq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-01-13T11:48:29.192 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:28 smithi088 bash[14069]: audit 2024-01-13T11:48:28.847957+0000 mon.smithi088 (mon.0) 184 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:29.193 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:28 smithi088 bash[14069]: audit 2024-01-13T11:48:28.848365+0000 mon.smithi088 (mon.0) 185 : audit [DBG] from='mgr.? 172.21.15.138:0/2380498778' entity='mgr.smithi138.divzmq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi138.divzmq/key"}]: dispatch 2024-01-13T11:48:29.193 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:28 smithi088 bash[14069]: audit 2024-01-13T11:48:28.848893+0000 mon.smithi088 (mon.0) 186 : audit [DBG] from='mgr.? 172.21.15.138:0/2380498778' entity='mgr.smithi138.divzmq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-01-13T11:48:29.193 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-01-13T11:48:29.193 INFO:teuthology.orchestra.run.smithi138.stdout:{"epoch":2,"fsid":"236db248-b209-11ee-95ac-87774f69a715","modified":"2024-01-13T11:48:23.766885Z","created":"2024-01-13T11:45:31.831492Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi088","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi138","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:3300","nonce":0},{"type":"v1","addr":"172.21.15.138:6789","nonce":0}]},"addr":"172.21.15.138:6789/0","public_addr":"172.21.15.138:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-01-13T11:48:29.194 INFO:teuthology.orchestra.run.smithi138.stderr:dumped monmap epoch 2 2024-01-13T11:48:29.223 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:28 smithi138 bash[17620]: cluster 2024-01-13T11:48:23.540050+0000 mgr.smithi088.xwlxdh (mgr.14182) 15 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:29.223 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:28 smithi138 bash[17620]: audit 2024-01-13T11:48:23.767521+0000 mon.smithi088 (mon.0) 166 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mon metadata", "id": "smithi138"}]: dispatch 2024-01-13T11:48:29.223 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:28 smithi138 bash[17620]: audit 2024-01-13T11:48:23.771706+0000 mon.smithi088 (mon.0) 167 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mon metadata", "id": "smithi088"}]: dispatch 2024-01-13T11:48:29.223 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:28 smithi138 bash[17620]: cluster 2024-01-13T11:48:23.771957+0000 mon.smithi088 (mon.0) 168 : cluster [INF] mon.smithi088 calling monitor election 2024-01-13T11:48:29.223 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:28 smithi138 bash[17620]: audit 2024-01-13T11:48:23.790500+0000 mon.smithi088 (mon.0) 169 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mon metadata", "id": "smithi138"}]: dispatch 2024-01-13T11:48:29.223 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:28 smithi138 bash[17620]: audit 2024-01-13T11:48:24.767016+0000 mon.smithi088 (mon.0) 170 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mon metadata", "id": "smithi138"}]: dispatch 2024-01-13T11:48:29.223 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:28 smithi138 bash[17620]: audit 2024-01-13T11:48:25.017175+0000 mon.smithi088 (mon.0) 171 : audit [DBG] from='mgr.? 172.21.15.138:0/2380498778' entity='mgr.smithi138.divzmq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi138.divzmq/crt"}]: dispatch 2024-01-13T11:48:29.223 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:28 smithi138 bash[17620]: cluster 2024-01-13T11:48:25.540541+0000 mgr.smithi088.xwlxdh (mgr.14182) 16 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:29.223 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:28 smithi138 bash[17620]: audit 2024-01-13T11:48:25.767301+0000 mon.smithi088 (mon.0) 172 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mon metadata", "id": "smithi138"}]: dispatch 2024-01-13T11:48:29.223 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:28 smithi138 bash[17620]: cluster 2024-01-13T11:48:25.768960+0000 mon.smithi138 (mon.1) 1 : cluster [INF] mon.smithi138 calling monitor election 2024-01-13T11:48:29.223 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:28 smithi138 bash[17620]: audit 2024-01-13T11:48:26.767349+0000 mon.smithi088 (mon.0) 173 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mon metadata", "id": "smithi138"}]: dispatch 2024-01-13T11:48:29.224 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:28 smithi138 bash[17620]: cluster 2024-01-13T11:48:27.541045+0000 mgr.smithi088.xwlxdh (mgr.14182) 17 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:29.224 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:28 smithi138 bash[17620]: audit 2024-01-13T11:48:27.768064+0000 mon.smithi088 (mon.0) 174 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mon metadata", "id": "smithi138"}]: dispatch 2024-01-13T11:48:29.224 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:28 smithi138 bash[17620]: audit 2024-01-13T11:48:28.767768+0000 mon.smithi088 (mon.0) 175 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mon metadata", "id": "smithi138"}]: dispatch 2024-01-13T11:48:29.224 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:28 smithi138 bash[17620]: cluster 2024-01-13T11:48:28.810800+0000 mon.smithi088 (mon.0) 176 : cluster [INF] mon.smithi088 is new leader, mons smithi088,smithi138 in quorum (ranks 0,1) 2024-01-13T11:48:29.224 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:28 smithi138 bash[17620]: cluster 2024-01-13T11:48:28.816108+0000 mon.smithi088 (mon.0) 177 : cluster [DBG] monmap e2: 2 mons at {smithi088=[v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0],smithi138=[v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0]} removed_ranks: {} 2024-01-13T11:48:29.224 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:28 smithi138 bash[17620]: cluster 2024-01-13T11:48:28.834235+0000 mon.smithi088 (mon.0) 178 : cluster [DBG] fsmap 2024-01-13T11:48:29.224 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:28 smithi138 bash[17620]: cluster 2024-01-13T11:48:28.834335+0000 mon.smithi088 (mon.0) 179 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-01-13T11:48:29.224 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:28 smithi138 bash[17620]: cluster 2024-01-13T11:48:28.834755+0000 mon.smithi088 (mon.0) 180 : cluster [DBG] mgrmap e17: smithi088.xwlxdh(active, since 29s) 2024-01-13T11:48:29.224 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:28 smithi138 bash[17620]: cluster 2024-01-13T11:48:28.836847+0000 mon.smithi088 (mon.0) 181 : cluster [DBG] Standby manager daemon smithi138.divzmq started 2024-01-13T11:48:29.224 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:28 smithi138 bash[17620]: cluster 2024-01-13T11:48:28.843724+0000 mon.smithi088 (mon.0) 182 : cluster [INF] overall HEALTH_OK 2024-01-13T11:48:29.224 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:28 smithi138 bash[17620]: audit 2024-01-13T11:48:28.844446+0000 mon.smithi088 (mon.0) 183 : audit [DBG] from='mgr.? 172.21.15.138:0/2380498778' entity='mgr.smithi138.divzmq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-01-13T11:48:29.224 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:28 smithi138 bash[17620]: audit 2024-01-13T11:48:28.847957+0000 mon.smithi088 (mon.0) 184 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:29.225 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:28 smithi138 bash[17620]: audit 2024-01-13T11:48:28.848365+0000 mon.smithi088 (mon.0) 185 : audit [DBG] from='mgr.? 172.21.15.138:0/2380498778' entity='mgr.smithi138.divzmq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi138.divzmq/key"}]: dispatch 2024-01-13T11:48:29.225 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:28 smithi138 bash[17620]: audit 2024-01-13T11:48:28.848893+0000 mon.smithi088 (mon.0) 186 : audit [DBG] from='mgr.? 172.21.15.138:0/2380498778' entity='mgr.smithi138.divzmq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-01-13T11:48:30.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:29 smithi138 bash[17620]: cluster 2024-01-13T11:48:28.903099+0000 mon.smithi088 (mon.0) 187 : cluster [DBG] mgrmap e18: smithi088.xwlxdh(active, since 29s), standbys: smithi138.divzmq 2024-01-13T11:48:30.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:29 smithi138 bash[17620]: audit 2024-01-13T11:48:28.903295+0000 mon.smithi088 (mon.0) 188 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mgr metadata", "who": "smithi138.divzmq", "id": "smithi138.divzmq"}]: dispatch 2024-01-13T11:48:30.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:29 smithi138 bash[17620]: audit 2024-01-13T11:48:29.199882+0000 mon.smithi088 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.138:0/208119459' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:48:30.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:29 smithi138 bash[17620]: audit 2024-01-13T11:48:29.768034+0000 mon.smithi088 (mon.0) 190 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mon metadata", "id": "smithi138"}]: dispatch 2024-01-13T11:48:30.392 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:29 smithi088 bash[14069]: cluster 2024-01-13T11:48:28.903099+0000 mon.smithi088 (mon.0) 187 : cluster [DBG] mgrmap e18: smithi088.xwlxdh(active, since 29s), standbys: smithi138.divzmq 2024-01-13T11:48:30.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:29 smithi088 bash[14069]: audit 2024-01-13T11:48:28.903295+0000 mon.smithi088 (mon.0) 188 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mgr metadata", "who": "smithi138.divzmq", "id": "smithi138.divzmq"}]: dispatch 2024-01-13T11:48:30.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:29 smithi088 bash[14069]: audit 2024-01-13T11:48:29.199882+0000 mon.smithi088 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.138:0/208119459' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T11:48:30.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:29 smithi088 bash[14069]: audit 2024-01-13T11:48:29.768034+0000 mon.smithi088 (mon.0) 190 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mon metadata", "id": "smithi138"}]: dispatch 2024-01-13T11:48:31.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:30 smithi138 bash[17620]: cluster 2024-01-13T11:48:29.541560+0000 mgr.smithi088.xwlxdh (mgr.14182) 18 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:31.392 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:30 smithi088 bash[14069]: cluster 2024-01-13T11:48:29.541560+0000 mgr.smithi088.xwlxdh (mgr.14182) 18 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:31.578 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-01-13T11:48:31.578 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph config generate-minimal-conf 2024-01-13T11:48:33.132 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:32 smithi088 bash[14069]: cluster 2024-01-13T11:48:31.541986+0000 mgr.smithi088.xwlxdh (mgr.14182) 19 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:33.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:32 smithi138 bash[17620]: cluster 2024-01-13T11:48:31.541986+0000 mgr.smithi088.xwlxdh (mgr.14182) 19 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:34.821 INFO:teuthology.orchestra.run.smithi088.stdout:# minimal ceph.conf for 236db248-b209-11ee-95ac-87774f69a715 2024-01-13T11:48:34.821 INFO:teuthology.orchestra.run.smithi088.stdout:[global] 2024-01-13T11:48:34.821 INFO:teuthology.orchestra.run.smithi088.stdout: fsid = 236db248-b209-11ee-95ac-87774f69a715 2024-01-13T11:48:34.821 INFO:teuthology.orchestra.run.smithi088.stdout: mon_host = [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] 2024-01-13T11:48:35.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:34 smithi138 bash[17620]: cluster 2024-01-13T11:48:33.542451+0000 mgr.smithi088.xwlxdh (mgr.14182) 20 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:35.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:34 smithi138 bash[17620]: audit 2024-01-13T11:48:34.830269+0000 mon.smithi088 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.88:0/900670440' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:48:35.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:34 smithi088 bash[14069]: cluster 2024-01-13T11:48:33.542451+0000 mgr.smithi088.xwlxdh (mgr.14182) 20 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:35.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:34 smithi088 bash[14069]: audit 2024-01-13T11:48:34.830269+0000 mon.smithi088 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.88:0/900670440' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:48:35.571 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-01-13T11:48:35.571 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-01-13T11:48:35.571 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/ceph/ceph.conf 2024-01-13T11:48:35.582 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-01-13T11:48:35.582 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-13T11:48:35.637 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-01-13T11:48:35.637 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/ceph/ceph.conf 2024-01-13T11:48:35.650 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-01-13T11:48:35.651 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-13T11:48:35.712 INFO:tasks.cephadm:Deploying OSDs... 2024-01-13T11:48:35.712 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-01-13T11:48:35.712 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/scratch_devs of=/dev/stdout 2024-01-13T11:48:35.720 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-01-13T11:48:35.720 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/nvme1n1 2024-01-13T11:48:35.770 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/nvme1n1 2024-01-13T11:48:35.770 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-13T11:48:35.770 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 526 Links: 1 Device type: 103,2 2024-01-13T11:48:35.770 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-13T11:48:35.770 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-01-13 11:47:00.486340410 +0000 2024-01-13T11:48:35.770 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-01-13 11:44:39.438748910 +0000 2024-01-13T11:48:35.770 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-01-13 11:44:39.438748910 +0000 2024-01-13T11:48:35.770 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: - 2024-01-13T11:48:35.770 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-01-13T11:48:35.825 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-01-13T11:48:35.825 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-01-13T11:48:35.825 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000458064 s, 1.1 MB/s 2024-01-13T11:48:35.826 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-01-13T11:48:35.875 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/nvme2n1 2024-01-13T11:48:35.926 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/nvme2n1 2024-01-13T11:48:35.926 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-13T11:48:35.926 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 534 Links: 1 Device type: 103,4 2024-01-13T11:48:35.926 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-13T11:48:35.926 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-01-13 11:47:00.506339949 +0000 2024-01-13T11:48:35.926 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-01-13 11:44:39.554688236 +0000 2024-01-13T11:48:35.926 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-01-13 11:44:39.554688236 +0000 2024-01-13T11:48:35.926 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: - 2024-01-13T11:48:35.926 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-01-13T11:48:35.981 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-01-13T11:48:35.982 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-01-13T11:48:35.982 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000501485 s, 1.0 MB/s 2024-01-13T11:48:35.982 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-01-13T11:48:36.036 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/nvme3n1 2024-01-13T11:48:36.090 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/nvme3n1 2024-01-13T11:48:36.090 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-13T11:48:36.090 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 542 Links: 1 Device type: 103,6 2024-01-13T11:48:36.090 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-13T11:48:36.090 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-01-13 11:47:00.526339489 +0000 2024-01-13T11:48:36.090 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-01-13 11:44:39.670627563 +0000 2024-01-13T11:48:36.090 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-01-13 11:44:39.670627563 +0000 2024-01-13T11:48:36.091 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: - 2024-01-13T11:48:36.091 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-01-13T11:48:36.144 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-01-13T11:48:36.144 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-01-13T11:48:36.144 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000473781 s, 1.1 MB/s 2024-01-13T11:48:36.145 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-01-13T11:48:36.196 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/nvme4n1 2024-01-13T11:48:36.246 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/nvme4n1 2024-01-13T11:48:36.246 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-13T11:48:36.246 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 550 Links: 1 Device type: 103,8 2024-01-13T11:48:36.246 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-13T11:48:36.246 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-01-13 11:47:00.542339120 +0000 2024-01-13T11:48:36.246 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-01-13 11:44:39.790564796 +0000 2024-01-13T11:48:36.246 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-01-13 11:44:39.790564796 +0000 2024-01-13T11:48:36.247 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: - 2024-01-13T11:48:36.247 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-01-13T11:48:36.299 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-01-13T11:48:36.300 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-01-13T11:48:36.300 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000395299 s, 1.3 MB/s 2024-01-13T11:48:36.301 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-01-13T11:48:36.352 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-01-13T11:48:36.352 DEBUG:teuthology.orchestra.run.smithi138:> dd if=/scratch_devs of=/dev/stdout 2024-01-13T11:48:36.360 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-01-13T11:48:36.360 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/nvme1n1 2024-01-13T11:48:36.407 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/nvme1n1 2024-01-13T11:48:36.408 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-13T11:48:36.408 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 526 Links: 1 Device type: 103,2 2024-01-13T11:48:36.408 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-13T11:48:36.408 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2024-01-13 11:48:06.386894758 +0000 2024-01-13T11:48:36.408 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2024-01-13 11:44:40.815658830 +0000 2024-01-13T11:48:36.408 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2024-01-13 11:44:40.815658830 +0000 2024-01-13T11:48:36.408 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: - 2024-01-13T11:48:36.408 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-01-13T11:48:36.460 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2024-01-13T11:48:36.460 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2024-01-13T11:48:36.460 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000496786 s, 1.0 MB/s 2024-01-13T11:48:36.461 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-01-13T11:48:36.513 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/nvme2n1 2024-01-13T11:48:36.564 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/nvme2n1 2024-01-13T11:48:36.564 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-13T11:48:36.564 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 534 Links: 1 Device type: 103,4 2024-01-13T11:48:36.564 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-13T11:48:36.564 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2024-01-13 11:48:06.410894218 +0000 2024-01-13T11:48:36.565 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2024-01-13 11:44:40.927712247 +0000 2024-01-13T11:48:36.565 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2024-01-13 11:44:40.927712247 +0000 2024-01-13T11:48:36.565 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: - 2024-01-13T11:48:36.565 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-01-13T11:48:36.619 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2024-01-13T11:48:36.619 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2024-01-13T11:48:36.619 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000468293 s, 1.1 MB/s 2024-01-13T11:48:36.620 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-01-13T11:48:36.674 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/nvme3n1 2024-01-13T11:48:36.727 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/nvme3n1 2024-01-13T11:48:36.727 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-13T11:48:36.727 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 542 Links: 1 Device type: 103,6 2024-01-13T11:48:36.728 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-13T11:48:36.728 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2024-01-13 11:48:06.430893767 +0000 2024-01-13T11:48:36.728 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2024-01-13 11:44:41.039748628 +0000 2024-01-13T11:48:36.728 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2024-01-13 11:44:41.039748628 +0000 2024-01-13T11:48:36.728 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: - 2024-01-13T11:48:36.728 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-01-13T11:48:36.785 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2024-01-13T11:48:36.785 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2024-01-13T11:48:36.785 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000330328 s, 1.5 MB/s 2024-01-13T11:48:36.786 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-01-13T11:48:36.837 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/nvme4n1 2024-01-13T11:48:36.889 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/nvme4n1 2024-01-13T11:48:36.889 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-13T11:48:36.889 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 550 Links: 1 Device type: 103,8 2024-01-13T11:48:36.889 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-13T11:48:36.889 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2024-01-13 11:48:06.450893317 +0000 2024-01-13T11:48:36.889 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2024-01-13 11:44:41.155748621 +0000 2024-01-13T11:48:36.889 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2024-01-13 11:44:41.155748621 +0000 2024-01-13T11:48:36.890 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: - 2024-01-13T11:48:36.890 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-01-13T11:48:36.942 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2024-01-13T11:48:36.942 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2024-01-13T11:48:36.942 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000443773 s, 1.2 MB/s 2024-01-13T11:48:36.943 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-01-13T11:48:36.997 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph orch apply osd --all-available-devices 2024-01-13T11:48:37.056 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:36 smithi138 bash[17620]: cluster 2024-01-13T11:48:35.542828+0000 mgr.smithi088.xwlxdh (mgr.14182) 21 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:37.056 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:36 smithi138 bash[17620]: audit 2024-01-13T11:48:36.725169+0000 mon.smithi088 (mon.0) 192 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:37.056 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:36 smithi138 bash[17620]: audit 2024-01-13T11:48:36.727693+0000 mon.smithi088 (mon.0) 193 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:48:37.392 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:36 smithi088 bash[14069]: cluster 2024-01-13T11:48:35.542828+0000 mgr.smithi088.xwlxdh (mgr.14182) 21 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:37.392 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:36 smithi088 bash[14069]: audit 2024-01-13T11:48:36.725169+0000 mon.smithi088 (mon.0) 192 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:37.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:36 smithi088 bash[14069]: audit 2024-01-13T11:48:36.727693+0000 mon.smithi088 (mon.0) 193 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:48:39.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:38 smithi138 bash[17620]: cluster 2024-01-13T11:48:37.543214+0000 mgr.smithi088.xwlxdh (mgr.14182) 22 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:39.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:38 smithi138 bash[17620]: audit 2024-01-13T11:48:38.857443+0000 mon.smithi088 (mon.0) 194 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:39.392 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:38 smithi088 bash[14069]: cluster 2024-01-13T11:48:37.543214+0000 mgr.smithi088.xwlxdh (mgr.14182) 22 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:39.392 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:38 smithi088 bash[14069]: audit 2024-01-13T11:48:38.857443+0000 mon.smithi088 (mon.0) 194 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:40.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:39 smithi138 bash[17620]: cluster 2024-01-13T11:48:39.543629+0000 mgr.smithi088.xwlxdh (mgr.14182) 23 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:40.392 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:39 smithi088 bash[14069]: cluster 2024-01-13T11:48:39.543629+0000 mgr.smithi088.xwlxdh (mgr.14182) 23 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:40.490 INFO:teuthology.orchestra.run.smithi138.stdout:Scheduled osd.all-available-devices update... 2024-01-13T11:48:41.276 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-01-13T11:48:41.276 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:48:41.625 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:41 smithi138 bash[17620]: audit 2024-01-13T11:48:40.488831+0000 mgr.smithi088.xwlxdh (mgr.14182) 24 : audit [DBG] from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply osd", "all_available_devices": true, "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T11:48:41.625 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:41 smithi138 bash[17620]: cephadm 2024-01-13T11:48:40.493149+0000 mgr.smithi088.xwlxdh (mgr.14182) 25 : cephadm [INF] Marking host: smithi088 for OSDSpec preview refresh. 2024-01-13T11:48:41.625 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:41 smithi138 bash[17620]: cephadm 2024-01-13T11:48:40.493343+0000 mgr.smithi088.xwlxdh (mgr.14182) 26 : cephadm [INF] Marking host: smithi138 for OSDSpec preview refresh. 2024-01-13T11:48:41.625 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:41 smithi138 bash[17620]: cephadm 2024-01-13T11:48:40.494097+0000 mgr.smithi088.xwlxdh (mgr.14182) 27 : cephadm [INF] Saving service osd.all-available-devices spec with placement * 2024-01-13T11:48:41.626 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:41 smithi138 bash[17620]: audit 2024-01-13T11:48:40.499241+0000 mon.smithi088 (mon.0) 195 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:41.642 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:41 smithi088 bash[14069]: audit 2024-01-13T11:48:40.488831+0000 mgr.smithi088.xwlxdh (mgr.14182) 24 : audit [DBG] from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply osd", "all_available_devices": true, "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T11:48:41.642 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:41 smithi088 bash[14069]: cephadm 2024-01-13T11:48:40.493149+0000 mgr.smithi088.xwlxdh (mgr.14182) 25 : cephadm [INF] Marking host: smithi088 for OSDSpec preview refresh. 2024-01-13T11:48:41.642 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:41 smithi088 bash[14069]: cephadm 2024-01-13T11:48:40.493343+0000 mgr.smithi088.xwlxdh (mgr.14182) 26 : cephadm [INF] Marking host: smithi138 for OSDSpec preview refresh. 2024-01-13T11:48:41.643 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:41 smithi088 bash[14069]: cephadm 2024-01-13T11:48:40.494097+0000 mgr.smithi088.xwlxdh (mgr.14182) 27 : cephadm [INF] Saving service osd.all-available-devices spec with placement * 2024-01-13T11:48:41.643 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:41 smithi088 bash[14069]: audit 2024-01-13T11:48:40.499241+0000 mon.smithi088 (mon.0) 195 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:42.847 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:42 smithi088 bash[14069]: cluster 2024-01-13T11:48:41.544066+0000 mgr.smithi088.xwlxdh (mgr.14182) 28 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:42.875 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:42 smithi138 bash[17620]: cluster 2024-01-13T11:48:41.544066+0000 mgr.smithi088.xwlxdh (mgr.14182) 28 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:44.554 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:48:44.838 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:44 smithi088 bash[14069]: audit 2024-01-13T11:48:43.533563+0000 mon.smithi088 (mon.0) 196 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:44.838 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:44 smithi088 bash[14069]: audit 2024-01-13T11:48:43.539905+0000 mon.smithi088 (mon.0) 197 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:44.838 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:44 smithi088 bash[14069]: audit 2024-01-13T11:48:43.541111+0000 mon.smithi088 (mon.0) 198 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:48:44.838 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:44 smithi088 bash[14069]: audit 2024-01-13T11:48:43.542672+0000 mon.smithi088 (mon.0) 199 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T11:48:44.838 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:44 smithi088 bash[14069]: cluster 2024-01-13T11:48:43.544406+0000 mgr.smithi088.xwlxdh (mgr.14182) 29 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:44.838 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:44 smithi088 bash[14069]: cephadm 2024-01-13T11:48:43.544717+0000 mgr.smithi088.xwlxdh (mgr.14182) 30 : cephadm [INF] Updating smithi088:/etc/ceph/ceph.conf 2024-01-13T11:48:44.838 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:44 smithi088 bash[14069]: cephadm 2024-01-13T11:48:43.545152+0000 mgr.smithi088.xwlxdh (mgr.14182) 31 : cephadm [INF] Updating smithi138:/etc/ceph/ceph.conf 2024-01-13T11:48:44.838 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:44 smithi088 bash[14069]: audit 2024-01-13T11:48:43.571055+0000 mon.smithi088 (mon.0) 200 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:44.838 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:44 smithi088 bash[14069]: audit 2024-01-13T11:48:43.734983+0000 mon.smithi088 (mon.0) 201 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:44.839 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:44 smithi088 bash[14069]: audit 2024-01-13T11:48:43.741643+0000 mon.smithi088 (mon.0) 202 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:44.839 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:44 smithi088 bash[14069]: audit 2024-01-13T11:48:43.749621+0000 mon.smithi088 (mon.0) 203 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T11:48:44.839 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:44 smithi088 bash[14069]: audit 2024-01-13T11:48:43.758043+0000 mon.smithi088 (mon.0) 204 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:48:44.839 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:44 smithi088 bash[14069]: audit 2024-01-13T11:48:43.759843+0000 mon.smithi088 (mon.0) 205 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:48:44.839 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:44 smithi088 bash[14069]: audit 2024-01-13T11:48:43.760318+0000 mon.smithi088 (mon.0) 206 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:48:44.839 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:44 smithi088 bash[14069]: audit 2024-01-13T11:48:43.761139+0000 mon.smithi088 (mon.0) 207 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:48:44.875 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:44 smithi138 bash[17620]: audit 2024-01-13T11:48:43.533563+0000 mon.smithi088 (mon.0) 196 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:44.875 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:44 smithi138 bash[17620]: audit 2024-01-13T11:48:43.539905+0000 mon.smithi088 (mon.0) 197 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:44.875 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:44 smithi138 bash[17620]: audit 2024-01-13T11:48:43.541111+0000 mon.smithi088 (mon.0) 198 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:48:44.876 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:44 smithi138 bash[17620]: audit 2024-01-13T11:48:43.542672+0000 mon.smithi088 (mon.0) 199 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T11:48:44.876 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:44 smithi138 bash[17620]: cluster 2024-01-13T11:48:43.544406+0000 mgr.smithi088.xwlxdh (mgr.14182) 29 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:44.876 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:44 smithi138 bash[17620]: cephadm 2024-01-13T11:48:43.544717+0000 mgr.smithi088.xwlxdh (mgr.14182) 30 : cephadm [INF] Updating smithi088:/etc/ceph/ceph.conf 2024-01-13T11:48:44.876 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:44 smithi138 bash[17620]: cephadm 2024-01-13T11:48:43.545152+0000 mgr.smithi088.xwlxdh (mgr.14182) 31 : cephadm [INF] Updating smithi138:/etc/ceph/ceph.conf 2024-01-13T11:48:44.876 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:44 smithi138 bash[17620]: audit 2024-01-13T11:48:43.571055+0000 mon.smithi088 (mon.0) 200 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:44.876 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:44 smithi138 bash[17620]: audit 2024-01-13T11:48:43.734983+0000 mon.smithi088 (mon.0) 201 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:44.876 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:44 smithi138 bash[17620]: audit 2024-01-13T11:48:43.741643+0000 mon.smithi088 (mon.0) 202 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:44.876 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:44 smithi138 bash[17620]: audit 2024-01-13T11:48:43.749621+0000 mon.smithi088 (mon.0) 203 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T11:48:44.876 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:44 smithi138 bash[17620]: audit 2024-01-13T11:48:43.758043+0000 mon.smithi088 (mon.0) 204 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:48:44.876 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:44 smithi138 bash[17620]: audit 2024-01-13T11:48:43.759843+0000 mon.smithi088 (mon.0) 205 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:48:44.876 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:44 smithi138 bash[17620]: audit 2024-01-13T11:48:43.760318+0000 mon.smithi088 (mon.0) 206 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:48:44.876 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:44 smithi138 bash[17620]: audit 2024-01-13T11:48:43.761139+0000 mon.smithi088 (mon.0) 207 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:48:45.389 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2024-01-13T11:48:45.642 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:45 smithi088 bash[14069]: audit 2024-01-13T11:48:44.563668+0000 mon.smithi088 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.88:0/1040436074' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:48:45.875 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:45 smithi138 bash[17620]: audit 2024-01-13T11:48:44.563668+0000 mon.smithi088 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.88:0/1040436074' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:48:46.389 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:48:46.875 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:46 smithi138 bash[17620]: cluster 2024-01-13T11:48:45.544851+0000 mgr.smithi088.xwlxdh (mgr.14182) 32 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:46.895 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:46 smithi088 bash[14069]: cluster 2024-01-13T11:48:45.544851+0000 mgr.smithi088.xwlxdh (mgr.14182) 32 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:49.098 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:48 smithi088 bash[14069]: cluster 2024-01-13T11:48:47.545273+0000 mgr.smithi088.xwlxdh (mgr.14182) 33 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:49.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:48 smithi138 bash[17620]: cluster 2024-01-13T11:48:47.545273+0000 mgr.smithi088.xwlxdh (mgr.14182) 33 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:50.006 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:48:51.123 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:50 smithi088 bash[14069]: cluster 2024-01-13T11:48:49.545641+0000 mgr.smithi088.xwlxdh (mgr.14182) 34 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:51.123 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:50 smithi088 bash[14069]: audit 2024-01-13T11:48:50.016318+0000 mon.smithi088 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.88:0/974315944' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:48:51.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:50 smithi138 bash[17620]: cluster 2024-01-13T11:48:49.545641+0000 mgr.smithi088.xwlxdh (mgr.14182) 34 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:51.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:50 smithi138 bash[17620]: audit 2024-01-13T11:48:50.016318+0000 mon.smithi088 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.88:0/974315944' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:48:51.302 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2024-01-13T11:48:52.302 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:48:52.875 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:52 smithi138 bash[17620]: cluster 2024-01-13T11:48:51.546051+0000 mgr.smithi088.xwlxdh (mgr.14182) 35 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:52.875 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:52 smithi138 bash[17620]: audit 2024-01-13T11:48:52.672291+0000 mon.smithi138 (mon.1) 2 : audit [INF] from='client.? 172.21.15.138:0/504268896' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "96ffa1e9-0d1c-4900-b3e6-8e112fff4f0f"}]: dispatch 2024-01-13T11:48:52.875 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:52 smithi138 bash[17620]: audit 2024-01-13T11:48:52.673584+0000 mon.smithi088 (mon.0) 210 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "96ffa1e9-0d1c-4900-b3e6-8e112fff4f0f"}]: dispatch 2024-01-13T11:48:52.875 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:52 smithi138 bash[17620]: audit 2024-01-13T11:48:52.682366+0000 mon.smithi088 (mon.0) 211 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "96ffa1e9-0d1c-4900-b3e6-8e112fff4f0f"}]': finished 2024-01-13T11:48:52.875 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:52 smithi138 bash[17620]: cluster 2024-01-13T11:48:52.682464+0000 mon.smithi088 (mon.0) 212 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-01-13T11:48:52.875 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:52 smithi138 bash[17620]: audit 2024-01-13T11:48:52.682618+0000 mon.smithi088 (mon.0) 213 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:48:53.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:52 smithi088 bash[14069]: cluster 2024-01-13T11:48:51.546051+0000 mgr.smithi088.xwlxdh (mgr.14182) 35 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:53.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:52 smithi088 bash[14069]: audit 2024-01-13T11:48:52.672291+0000 mon.smithi138 (mon.1) 2 : audit [INF] from='client.? 172.21.15.138:0/504268896' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "96ffa1e9-0d1c-4900-b3e6-8e112fff4f0f"}]: dispatch 2024-01-13T11:48:53.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:52 smithi088 bash[14069]: audit 2024-01-13T11:48:52.673584+0000 mon.smithi088 (mon.0) 210 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "96ffa1e9-0d1c-4900-b3e6-8e112fff4f0f"}]: dispatch 2024-01-13T11:48:53.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:52 smithi088 bash[14069]: audit 2024-01-13T11:48:52.682366+0000 mon.smithi088 (mon.0) 211 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "96ffa1e9-0d1c-4900-b3e6-8e112fff4f0f"}]': finished 2024-01-13T11:48:53.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:52 smithi088 bash[14069]: cluster 2024-01-13T11:48:52.682464+0000 mon.smithi088 (mon.0) 212 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-01-13T11:48:53.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:52 smithi088 bash[14069]: audit 2024-01-13T11:48:52.682618+0000 mon.smithi088 (mon.0) 213 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:48:54.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:53 smithi138 bash[17620]: audit 2024-01-13T11:48:53.032207+0000 mon.smithi138 (mon.1) 3 : audit [INF] from='client.? 172.21.15.138:0/4091467377' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:48:54.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:53 smithi138 bash[17620]: audit 2024-01-13T11:48:53.033069+0000 mon.smithi088 (mon.0) 214 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:48:54.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:53 smithi138 bash[17620]: audit 2024-01-13T11:48:53.041961+0000 mon.smithi088 (mon.0) 215 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:48:54.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:53 smithi138 bash[17620]: cluster 2024-01-13T11:48:53.042055+0000 mon.smithi088 (mon.0) 216 : cluster [DBG] osdmap e7: 0 total, 0 up, 0 in 2024-01-13T11:48:54.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:53 smithi138 bash[17620]: audit 2024-01-13T11:48:53.819650+0000 mon.smithi088 (mon.0) 217 : audit [INF] from='client.? 172.21.15.88:0/2663352042' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b24bc7d-31c0-405c-976b-0021aa4d7fea"}]: dispatch 2024-01-13T11:48:54.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:53 smithi138 bash[17620]: audit 2024-01-13T11:48:53.826281+0000 mon.smithi088 (mon.0) 218 : audit [INF] from='client.? 172.21.15.88:0/2663352042' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7b24bc7d-31c0-405c-976b-0021aa4d7fea"}]': finished 2024-01-13T11:48:54.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:53 smithi138 bash[17620]: cluster 2024-01-13T11:48:53.826384+0000 mon.smithi088 (mon.0) 219 : cluster [DBG] osdmap e8: 1 total, 0 up, 1 in 2024-01-13T11:48:54.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:53 smithi138 bash[17620]: audit 2024-01-13T11:48:53.826518+0000 mon.smithi088 (mon.0) 220 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:48:54.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:53 smithi088 bash[14069]: audit 2024-01-13T11:48:53.032207+0000 mon.smithi138 (mon.1) 3 : audit [INF] from='client.? 172.21.15.138:0/4091467377' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:48:54.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:53 smithi088 bash[14069]: audit 2024-01-13T11:48:53.033069+0000 mon.smithi088 (mon.0) 214 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:48:54.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:53 smithi088 bash[14069]: audit 2024-01-13T11:48:53.041961+0000 mon.smithi088 (mon.0) 215 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:48:54.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:53 smithi088 bash[14069]: cluster 2024-01-13T11:48:53.042055+0000 mon.smithi088 (mon.0) 216 : cluster [DBG] osdmap e7: 0 total, 0 up, 0 in 2024-01-13T11:48:54.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:53 smithi088 bash[14069]: audit 2024-01-13T11:48:53.819650+0000 mon.smithi088 (mon.0) 217 : audit [INF] from='client.? 172.21.15.88:0/2663352042' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b24bc7d-31c0-405c-976b-0021aa4d7fea"}]: dispatch 2024-01-13T11:48:54.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:53 smithi088 bash[14069]: audit 2024-01-13T11:48:53.826281+0000 mon.smithi088 (mon.0) 218 : audit [INF] from='client.? 172.21.15.88:0/2663352042' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7b24bc7d-31c0-405c-976b-0021aa4d7fea"}]': finished 2024-01-13T11:48:54.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:53 smithi088 bash[14069]: cluster 2024-01-13T11:48:53.826384+0000 mon.smithi088 (mon.0) 219 : cluster [DBG] osdmap e8: 1 total, 0 up, 1 in 2024-01-13T11:48:54.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:53 smithi088 bash[14069]: audit 2024-01-13T11:48:53.826518+0000 mon.smithi088 (mon.0) 220 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:48:55.102 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: cluster 2024-01-13T11:48:53.546330+0000 mgr.smithi088.xwlxdh (mgr.14182) 36 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:55.102 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: cephadm 2024-01-13T11:48:53.923298+0000 mgr.smithi088.xwlxdh (mgr.14182) 37 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:48:55.102 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:48:55.103 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:48:55.103 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: return f(*arg) 2024-01-13T11:48:55.103 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:48:55.103 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:48:55.103 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:48:55.103 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: code, '\n'.join(err))) 2024-01-13T11:48:55.103 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:48:55.103 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmppxxd1fjk:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpgvusnu2k:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:48:55.103 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:48:55.103 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:48:55.103 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:48:55.103 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 96ffa1e9-0d1c-4900-b3e6-8e112fff4f0f 2024-01-13T11:48:55.103 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:48:55.104 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:48:55.104 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:48:55.104 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:48:55.104 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:48:55.104 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:48:55.105 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:48:55.105 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:48:55.105 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:48:55.105 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:48:55.105 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:48:55.105 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:48:55.105 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:48:55.105 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:48:55.105 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:48:55.105 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:48:55.105 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:48:55.105 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:48:55.105 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:48:55.106 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:48:55.106 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:48:55.106 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:48:55.106 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:48:55.106 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:48:55.106 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:48:55.106 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:48:55.106 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:48:55.106 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:48:55.106 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:48:55.106 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:48:55.106 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:48:55.106 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:48:55.106 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:48:55.106 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:48:55.107 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:48:55.107 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:48:55.107 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:48:55.107 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:48:55.107 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:48:55.107 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:48:55.107 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:48:55.107 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:48:55.107 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:48:55.107 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:48:55.107 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:48:55.107 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:48:55.107 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:48:55.107 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:48:55.107 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:48:55.108 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:48:55.108 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:48:55.108 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:48:55.108 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:48:55.108 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:48:55.108 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:48:55.108 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:48:55.108 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:48:55.108 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:48:55.108 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: main() 2024-01-13T11:48:55.108 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:48:55.108 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:48:55.108 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:48:55.108 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:48:55.109 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:48:55.109 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:48:55.109 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:48:55.109 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:48:55.109 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:48:55.109 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:48:55.109 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:48:55.109 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:48:55.109 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:48:55.109 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:48:55.109 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmppxxd1fjk:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpgvusnu2k:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:48:55.109 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: audit 2024-01-13T11:48:54.153346+0000 mon.smithi088 (mon.0) 221 : audit [INF] from='client.? 172.21.15.88:0/256760998' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:48:55.109 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: audit 2024-01-13T11:48:54.159772+0000 mon.smithi088 (mon.0) 222 : audit [INF] from='client.? 172.21.15.88:0/256760998' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:48:55.109 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:54 smithi088 bash[14069]: cluster 2024-01-13T11:48:54.159888+0000 mon.smithi088 (mon.0) 223 : cluster [DBG] osdmap e9: 0 total, 0 up, 0 in 2024-01-13T11:48:55.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: cluster 2024-01-13T11:48:53.546330+0000 mgr.smithi088.xwlxdh (mgr.14182) 36 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:55.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: cephadm 2024-01-13T11:48:53.923298+0000 mgr.smithi088.xwlxdh (mgr.14182) 37 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:48:55.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:48:55.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:48:55.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: return f(*arg) 2024-01-13T11:48:55.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:48:55.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:48:55.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:48:55.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: code, '\n'.join(err))) 2024-01-13T11:48:55.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:48:55.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmppxxd1fjk:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpgvusnu2k:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:48:55.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:48:55.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:48:55.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:48:55.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 96ffa1e9-0d1c-4900-b3e6-8e112fff4f0f 2024-01-13T11:48:55.127 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:48:55.127 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:48:55.127 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:48:55.127 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:48:55.127 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:48:55.127 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:48:55.127 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:48:55.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:48:55.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:48:55.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:48:55.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:48:55.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:48:55.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:48:55.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:48:55.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:48:55.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:48:55.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:48:55.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:48:55.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:48:55.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:48:55.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:48:55.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:48:55.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:48:55.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:48:55.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:48:55.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:48:55.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:48:55.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:48:55.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:48:55.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:48:55.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:48:55.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:48:55.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:48:55.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:48:55.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:48:55.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:48:55.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:48:55.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:48:55.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:48:55.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:48:55.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:48:55.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:48:55.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:48:55.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:48:55.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:48:55.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:48:55.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:48:55.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:48:55.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:48:55.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:48:55.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:48:55.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:48:55.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:48:55.133 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:48:55.133 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:48:55.133 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:48:55.133 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:48:55.133 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:48:55.133 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: main() 2024-01-13T11:48:55.133 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:48:55.133 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:48:55.133 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:48:55.133 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:48:55.133 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:48:55.133 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:48:55.133 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:48:55.133 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:48:55.133 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:48:55.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:48:55.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:48:55.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:48:55.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:48:55.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:48:55.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmppxxd1fjk:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpgvusnu2k:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:48:55.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: audit 2024-01-13T11:48:54.153346+0000 mon.smithi088 (mon.0) 221 : audit [INF] from='client.? 172.21.15.88:0/256760998' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:48:55.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: audit 2024-01-13T11:48:54.159772+0000 mon.smithi088 (mon.0) 222 : audit [INF] from='client.? 172.21.15.88:0/256760998' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:48:55.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:54 smithi138 bash[17620]: cluster 2024-01-13T11:48:54.159888+0000 mon.smithi088 (mon.0) 223 : cluster [DBG] osdmap e9: 0 total, 0 up, 0 in 2024-01-13T11:48:56.258 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:48:57.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: cluster 2024-01-13T11:48:55.546705+0000 mgr.smithi088.xwlxdh (mgr.14182) 38 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:57.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: cephadm 2024-01-13T11:48:55.854672+0000 mgr.smithi088.xwlxdh (mgr.14182) 39 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:48:57.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:48:57.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:48:57.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: return f(*arg) 2024-01-13T11:48:57.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:48:57.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:48:57.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:48:57.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: code, '\n'.join(err))) 2024-01-13T11:48:57.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi088/config 2024-01-13T11:48:57.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpujfimqmv:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpvogw4h9k:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:48:57.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:48:57.127 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:48:57.127 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:48:57.127 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 7b24bc7d-31c0-405c-976b-0021aa4d7fea 2024-01-13T11:48:57.127 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:48:57.127 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:48:57.127 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:48:57.127 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:48:57.127 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:48:57.127 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:48:57.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:48:57.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:48:57.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:48:57.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:48:57.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:48:57.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:48:57.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:48:57.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:48:57.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:48:57.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:48:57.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:48:57.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:48:57.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:48:57.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:48:57.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:48:57.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:48:57.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:48:57.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:48:57.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:48:57.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:48:57.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:48:57.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:48:57.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:48:57.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:48:57.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:48:57.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:48:57.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:48:57.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:48:57.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:48:57.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:48:57.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:48:57.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:48:57.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:48:57.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:48:57.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:48:57.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:48:57.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:48:57.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:48:57.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:48:57.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:48:57.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:48:57.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:48:57.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:48:57.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:48:57.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:48:57.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:48:57.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:48:57.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:48:57.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:48:57.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:48:57.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:48:57.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:48:57.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: main() 2024-01-13T11:48:57.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:48:57.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:48:57.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:48:57.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:48:57.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:48:57.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:48:57.133 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:48:57.133 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:48:57.133 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:48:57.133 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:48:57.133 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:48:57.133 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:48:57.133 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:48:57.133 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:48:57.133 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpujfimqmv:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpvogw4h9k:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:48:57.133 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: cephadm 2024-01-13T11:48:55.858506+0000 mgr.smithi088.xwlxdh (mgr.14182) 40 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T11:48:57.133 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: service_id: all-available-devices 2024-01-13T11:48:57.133 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: service_name: osd.all-available-devices 2024-01-13T11:48:57.133 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: placement: 2024-01-13T11:48:57.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: host_pattern: '*' 2024-01-13T11:48:57.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: spec: 2024-01-13T11:48:57.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: data_devices: 2024-01-13T11:48:57.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: all: true 2024-01-13T11:48:57.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: filter_logic: AND 2024-01-13T11:48:57.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: objectstore: bluestore 2024-01-13T11:48:57.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:48:57.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmppxxd1fjk:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpgvusnu2k:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:48:57.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:48:57.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:48:57.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:48:57.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 96ffa1e9-0d1c-4900-b3e6-8e112fff4f0f 2024-01-13T11:48:57.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:48:57.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:48:57.135 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:48:57.135 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:48:57.135 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:48:57.135 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:48:57.135 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:48:57.135 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:48:57.135 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:48:57.135 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:48:57.135 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:48:57.135 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:48:57.135 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:48:57.135 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:48:57.135 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:48:57.136 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:48:57.136 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:48:57.136 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:48:57.136 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:48:57.136 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:48:57.136 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:48:57.136 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:48:57.136 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:48:57.136 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:48:57.136 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:48:57.136 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:48:57.136 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:48:57.137 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:48:57.137 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:48:57.137 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:48:57.137 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:48:57.137 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:48:57.137 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:48:57.137 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:48:57.137 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:48:57.137 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:48:57.137 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:48:57.137 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:48:57.137 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:48:57.137 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:48:57.138 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:48:57.138 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:48:57.138 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:48:57.138 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:48:57.138 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:48:57.138 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:48:57.138 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:48:57.138 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:48:57.138 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:48:57.138 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:48:57.138 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:48:57.138 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:48:57.138 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:48:57.139 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:48:57.139 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:48:57.139 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:48:57.139 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:48:57.139 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:48:57.139 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: main() 2024-01-13T11:48:57.139 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:48:57.139 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:48:57.139 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:48:57.139 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:48:57.139 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:48:57.139 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:48:57.140 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:48:57.140 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:48:57.140 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:48:57.141 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:48:57.141 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:48:57.141 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:48:57.141 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:48:57.141 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:48:57.141 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmppxxd1fjk:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpgvusnu2k:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:48:57.141 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:48:57.141 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T11:48:57.141 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: if self._apply_service(spec): 2024-01-13T11:48:57.141 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T11:48:57.141 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T11:48:57.141 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T11:48:57.141 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T11:48:57.142 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T11:48:57.142 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T11:48:57.142 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T11:48:57.142 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T11:48:57.142 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T11:48:57.142 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: raise self._value 2024-01-13T11:48:57.142 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T11:48:57.142 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: result = (True, func(*args, **kwds)) 2024-01-13T11:48:57.142 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T11:48:57.142 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: return list(map(*args)) 2024-01-13T11:48:57.142 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:48:57.142 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: return f(*arg) 2024-01-13T11:48:57.142 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:48:57.142 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:48:57.143 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:48:57.143 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: code, '\n'.join(err))) 2024-01-13T11:48:57.143 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:48:57.143 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmppxxd1fjk:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpgvusnu2k:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:48:57.143 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:48:57.143 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:48:57.143 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:48:57.143 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 96ffa1e9-0d1c-4900-b3e6-8e112fff4f0f 2024-01-13T11:48:57.143 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:48:57.143 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:48:57.143 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:48:57.143 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:48:57.143 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:48:57.144 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:48:57.144 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:48:57.144 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:48:57.144 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:48:57.144 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:48:57.144 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:48:57.144 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:48:57.144 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:48:57.144 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:48:57.144 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:48:57.144 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:48:57.144 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:48:57.144 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:48:57.144 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:48:57.145 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:48:57.145 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:48:57.145 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:48:57.145 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:48:57.145 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:48:57.145 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:48:57.145 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:48:57.145 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:48:57.145 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:48:57.145 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:48:57.145 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:48:57.145 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:48:57.146 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:48:57.146 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:48:57.146 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:48:57.146 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:48:57.146 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:48:57.146 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:48:57.146 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:48:57.146 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:48:57.146 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:48:57.146 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:48:57.146 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:48:57.146 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:48:57.146 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:48:57.146 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:48:57.147 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:48:57.147 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:48:57.147 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:48:57.147 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:48:57.147 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:48:57.147 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:48:57.147 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:48:57.147 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:48:57.147 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:48:57.147 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:48:57.147 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:48:57.147 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:48:57.147 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:48:57.148 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: main() 2024-01-13T11:48:57.148 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:48:57.148 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:48:57.148 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:48:57.148 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:48:57.148 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:48:57.148 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:48:57.148 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:48:57.148 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:48:57.148 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:48:57.148 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:48:57.148 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:48:57.148 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:48:57.149 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:48:57.149 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:48:57.149 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmppxxd1fjk:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpgvusnu2k:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:48:57.149 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: cephadm 2024-01-13T11:48:55.859771+0000 mgr.smithi088.xwlxdh (mgr.14182) 41 : cephadm [INF] Reconfiguring mon.smithi088 (unknown last config time)... 2024-01-13T11:48:57.149 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: cluster 2024-01-13T11:48:55.860138+0000 mgr.smithi088.xwlxdh (mgr.14182) 42 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:57.149 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: audit 2024-01-13T11:48:55.860302+0000 mon.smithi088 (mon.0) 224 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-13T11:48:57.149 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: audit 2024-01-13T11:48:55.861368+0000 mon.smithi088 (mon.0) 225 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-13T11:48:57.149 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: audit 2024-01-13T11:48:55.862449+0000 mon.smithi088 (mon.0) 226 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:48:57.149 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: cephadm 2024-01-13T11:48:55.863482+0000 mgr.smithi088.xwlxdh (mgr.14182) 43 : cephadm [INF] Reconfiguring daemon mon.smithi088 on smithi088 2024-01-13T11:48:57.149 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:56 smithi138 bash[17620]: audit 2024-01-13T11:48:56.268360+0000 mon.smithi088 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.88:0/3586266608' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:48:57.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: cluster 2024-01-13T11:48:55.546705+0000 mgr.smithi088.xwlxdh (mgr.14182) 38 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:57.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: cephadm 2024-01-13T11:48:55.854672+0000 mgr.smithi088.xwlxdh (mgr.14182) 39 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:48:57.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:48:57.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:48:57.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: return f(*arg) 2024-01-13T11:48:57.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:48:57.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:48:57.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:48:57.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: code, '\n'.join(err))) 2024-01-13T11:48:57.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi088/config 2024-01-13T11:48:57.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpujfimqmv:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpvogw4h9k:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:48:57.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:48:57.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:48:57.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:48:57.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 7b24bc7d-31c0-405c-976b-0021aa4d7fea 2024-01-13T11:48:57.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:48:57.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:48:57.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:48:57.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:48:57.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:48:57.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:48:57.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:48:57.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:48:57.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:48:57.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:48:57.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:48:57.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:48:57.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:48:57.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:48:57.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:48:57.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:48:57.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:48:57.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:48:57.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:48:57.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:48:57.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:48:57.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:48:57.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:48:57.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:48:57.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:48:57.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:48:57.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:48:57.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:48:57.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:48:57.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:48:57.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:48:57.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:48:57.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:48:57.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:48:57.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:48:57.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:48:57.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:48:57.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:48:57.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:48:57.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:48:57.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:48:57.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:48:57.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:48:57.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:48:57.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:48:57.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:48:57.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:48:57.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:48:57.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:48:57.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:48:57.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:48:57.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:48:57.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:48:57.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:48:57.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:48:57.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:48:57.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:48:57.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:48:57.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: main() 2024-01-13T11:48:57.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:48:57.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:48:57.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:48:57.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:48:57.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:48:57.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:48:57.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:48:57.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:48:57.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:48:57.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:48:57.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:48:57.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:48:57.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:48:57.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:48:57.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpujfimqmv:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpvogw4h9k:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:48:57.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: cephadm 2024-01-13T11:48:55.858506+0000 mgr.smithi088.xwlxdh (mgr.14182) 40 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T11:48:57.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: service_id: all-available-devices 2024-01-13T11:48:57.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: service_name: osd.all-available-devices 2024-01-13T11:48:57.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: placement: 2024-01-13T11:48:57.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: host_pattern: '*' 2024-01-13T11:48:57.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: spec: 2024-01-13T11:48:57.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: data_devices: 2024-01-13T11:48:57.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: all: true 2024-01-13T11:48:57.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: filter_logic: AND 2024-01-13T11:48:57.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: objectstore: bluestore 2024-01-13T11:48:57.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:48:57.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmppxxd1fjk:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpgvusnu2k:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:48:57.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:48:57.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:48:57.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:48:57.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 96ffa1e9-0d1c-4900-b3e6-8e112fff4f0f 2024-01-13T11:48:57.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:48:57.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:48:57.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:48:57.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:48:57.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:48:57.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:48:57.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:48:57.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:48:57.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:48:57.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:48:57.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:48:57.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:48:57.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:48:57.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:48:57.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:48:57.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:48:57.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:48:57.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:48:57.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:48:57.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:48:57.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:48:57.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:48:57.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:48:57.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:48:57.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:48:57.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:48:57.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:48:57.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:48:57.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:48:57.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:48:57.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:48:57.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:48:57.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:48:57.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:48:57.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:48:57.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:48:57.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:48:57.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:48:57.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:48:57.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:48:57.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:48:57.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:48:57.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:48:57.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:48:57.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:48:57.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:48:57.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:48:57.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:48:57.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:48:57.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:48:57.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:48:57.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:48:57.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:48:57.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:48:57.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:48:57.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:48:57.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:48:57.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:48:57.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: main() 2024-01-13T11:48:57.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:48:57.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:48:57.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:48:57.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:48:57.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:48:57.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:48:57.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:48:57.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:48:57.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:48:57.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:48:57.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:48:57.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:48:57.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:48:57.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:48:57.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmppxxd1fjk:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpgvusnu2k:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:48:57.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:48:57.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T11:48:57.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: if self._apply_service(spec): 2024-01-13T11:48:57.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T11:48:57.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T11:48:57.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T11:48:57.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T11:48:57.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T11:48:57.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T11:48:57.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T11:48:57.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T11:48:57.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T11:48:57.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: raise self._value 2024-01-13T11:48:57.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T11:48:57.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: result = (True, func(*args, **kwds)) 2024-01-13T11:48:57.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T11:48:57.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: return list(map(*args)) 2024-01-13T11:48:57.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:48:57.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: return f(*arg) 2024-01-13T11:48:57.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:48:57.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:48:57.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:48:57.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: code, '\n'.join(err))) 2024-01-13T11:48:57.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:48:57.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmppxxd1fjk:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpgvusnu2k:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:48:57.166 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:48:57.166 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:48:57.166 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:48:57.166 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 96ffa1e9-0d1c-4900-b3e6-8e112fff4f0f 2024-01-13T11:48:57.166 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:48:57.166 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:48:57.166 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:48:57.166 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:48:57.166 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:48:57.166 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:48:57.166 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:48:57.166 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:48:57.167 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:48:57.167 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:48:57.167 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:48:57.167 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:48:57.167 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:48:57.167 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:48:57.167 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:48:57.167 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:48:57.167 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:48:57.167 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:48:57.167 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:48:57.167 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:48:57.167 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:48:57.168 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:48:57.168 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:48:57.168 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:48:57.168 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:48:57.168 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:48:57.168 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:48:57.168 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:48:57.168 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:48:57.168 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:48:57.168 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:48:57.168 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:48:57.168 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:48:57.168 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:48:57.169 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:48:57.169 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:48:57.169 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:48:57.169 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:48:57.169 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:48:57.169 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:48:57.169 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:48:57.169 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:48:57.169 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:48:57.169 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:48:57.169 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:48:57.169 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:48:57.169 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:48:57.170 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:48:57.170 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:48:57.170 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:48:57.170 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:48:57.170 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:48:57.170 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:48:57.170 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:48:57.170 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:48:57.170 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:48:57.170 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:48:57.170 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:48:57.170 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: main() 2024-01-13T11:48:57.170 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:48:57.170 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:48:57.171 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:48:57.171 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:48:57.171 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:48:57.171 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:48:57.171 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:48:57.171 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:48:57.171 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:48:57.171 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:48:57.171 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:48:57.171 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:48:57.171 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:48:57.171 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:48:57.171 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmppxxd1fjk:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpgvusnu2k:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:48:57.172 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: cephadm 2024-01-13T11:48:55.859771+0000 mgr.smithi088.xwlxdh (mgr.14182) 41 : cephadm [INF] Reconfiguring mon.smithi088 (unknown last config time)... 2024-01-13T11:48:57.172 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: cluster 2024-01-13T11:48:55.860138+0000 mgr.smithi088.xwlxdh (mgr.14182) 42 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:57.172 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: audit 2024-01-13T11:48:55.860302+0000 mon.smithi088 (mon.0) 224 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-13T11:48:57.172 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: audit 2024-01-13T11:48:55.861368+0000 mon.smithi088 (mon.0) 225 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-13T11:48:57.172 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: audit 2024-01-13T11:48:55.862449+0000 mon.smithi088 (mon.0) 226 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:48:57.172 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: cephadm 2024-01-13T11:48:55.863482+0000 mgr.smithi088.xwlxdh (mgr.14182) 43 : cephadm [INF] Reconfiguring daemon mon.smithi088 on smithi088 2024-01-13T11:48:57.172 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:56 smithi088 bash[14069]: audit 2024-01-13T11:48:56.268360+0000 mon.smithi088 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.88:0/3586266608' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:48:57.565 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":9,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:48:58.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:57 smithi138 bash[17620]: cluster 2024-01-13T11:48:56.846220+0000 mon.smithi088 (mon.0) 228 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T11:48:58.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:57 smithi088 bash[14069]: cluster 2024-01-13T11:48:56.846220+0000 mon.smithi088 (mon.0) 228 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T11:48:58.567 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:48:59.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:58 smithi138 bash[17620]: cluster 2024-01-13T11:48:57.861249+0000 mgr.smithi088.xwlxdh (mgr.14182) 44 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:59.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:58 smithi138 bash[17620]: audit 2024-01-13T11:48:58.536100+0000 mon.smithi088 (mon.0) 229 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:59.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:58 smithi138 bash[17620]: audit 2024-01-13T11:48:58.537518+0000 mon.smithi088 (mon.0) 230 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi088.xwlxdh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-13T11:48:59.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:58 smithi138 bash[17620]: audit 2024-01-13T11:48:58.539077+0000 mon.smithi088 (mon.0) 231 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-13T11:48:59.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:58 smithi138 bash[17620]: audit 2024-01-13T11:48:58.540342+0000 mon.smithi088 (mon.0) 232 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:48:59.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:58 smithi088 bash[14069]: cluster 2024-01-13T11:48:57.861249+0000 mgr.smithi088.xwlxdh (mgr.14182) 44 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:48:59.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:58 smithi088 bash[14069]: audit 2024-01-13T11:48:58.536100+0000 mon.smithi088 (mon.0) 229 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:48:59.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:58 smithi088 bash[14069]: audit 2024-01-13T11:48:58.537518+0000 mon.smithi088 (mon.0) 230 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi088.xwlxdh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-13T11:48:59.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:58 smithi088 bash[14069]: audit 2024-01-13T11:48:58.539077+0000 mon.smithi088 (mon.0) 231 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-13T11:48:59.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:58 smithi088 bash[14069]: audit 2024-01-13T11:48:58.540342+0000 mon.smithi088 (mon.0) 232 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:49:00.125 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:59 smithi088 bash[14069]: cephadm 2024-01-13T11:48:58.537065+0000 mgr.smithi088.xwlxdh (mgr.14182) 45 : cephadm [INF] Reconfiguring mgr.smithi088.xwlxdh (unknown last config time)... 2024-01-13T11:49:00.125 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:59 smithi088 bash[14069]: cephadm 2024-01-13T11:48:58.541499+0000 mgr.smithi088.xwlxdh (mgr.14182) 46 : cephadm [INF] Reconfiguring daemon mgr.smithi088.xwlxdh on smithi088 2024-01-13T11:49:00.125 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:48:59 smithi088 bash[14069]: audit 2024-01-13T11:48:59.724886+0000 mon.smithi088 (mon.0) 233 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:49:00.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:59 smithi138 bash[17620]: cephadm 2024-01-13T11:48:58.537065+0000 mgr.smithi088.xwlxdh (mgr.14182) 45 : cephadm [INF] Reconfiguring mgr.smithi088.xwlxdh (unknown last config time)... 2024-01-13T11:49:00.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:59 smithi138 bash[17620]: cephadm 2024-01-13T11:48:58.541499+0000 mgr.smithi088.xwlxdh (mgr.14182) 46 : cephadm [INF] Reconfiguring daemon mgr.smithi088.xwlxdh on smithi088 2024-01-13T11:49:00.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:48:59 smithi138 bash[17620]: audit 2024-01-13T11:48:59.724886+0000 mon.smithi088 (mon.0) 233 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:49:01.122 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:00 smithi088 bash[14069]: cluster 2024-01-13T11:48:59.862542+0000 mgr.smithi088.xwlxdh (mgr.14182) 47 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:01.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:00 smithi138 bash[17620]: cluster 2024-01-13T11:48:59.862542+0000 mgr.smithi088.xwlxdh (mgr.14182) 47 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:02.642 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:02 smithi088 bash[14069]: audit 2024-01-13T11:49:01.394989+0000 mon.smithi088 (mon.0) 234 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:49:02.642 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:02 smithi088 bash[14069]: cephadm 2024-01-13T11:49:01.396142+0000 mgr.smithi088.xwlxdh (mgr.14182) 48 : cephadm [INF] Reconfiguring alertmanager.smithi088 (dependencies changed)... 2024-01-13T11:49:02.643 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:02 smithi088 bash[14069]: cephadm 2024-01-13T11:49:01.405918+0000 mgr.smithi088.xwlxdh (mgr.14182) 49 : cephadm [INF] Reconfiguring daemon alertmanager.smithi088 on smithi088 2024-01-13T11:49:02.643 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:02 smithi088 bash[14069]: cluster 2024-01-13T11:49:01.863224+0000 mgr.smithi088.xwlxdh (mgr.14182) 50 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:02.875 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:02 smithi138 bash[17620]: audit 2024-01-13T11:49:01.394989+0000 mon.smithi088 (mon.0) 234 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:49:02.875 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:02 smithi138 bash[17620]: cephadm 2024-01-13T11:49:01.396142+0000 mgr.smithi088.xwlxdh (mgr.14182) 48 : cephadm [INF] Reconfiguring alertmanager.smithi088 (dependencies changed)... 2024-01-13T11:49:02.875 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:02 smithi138 bash[17620]: cephadm 2024-01-13T11:49:01.405918+0000 mgr.smithi088.xwlxdh (mgr.14182) 49 : cephadm [INF] Reconfiguring daemon alertmanager.smithi088 on smithi088 2024-01-13T11:49:02.875 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:02 smithi138 bash[17620]: cluster 2024-01-13T11:49:01.863224+0000 mgr.smithi088.xwlxdh (mgr.14182) 50 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:03.299 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:49:04.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:03 smithi138 bash[17620]: audit 2024-01-13T11:49:03.308882+0000 mon.smithi088 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.88:0/621135166' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:49:04.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:03 smithi088 bash[14069]: audit 2024-01-13T11:49:03.308882+0000 mon.smithi088 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.88:0/621135166' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:49:04.734 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":9,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:49:05.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:04 smithi138 bash[17620]: cluster 2024-01-13T11:49:03.864163+0000 mgr.smithi088.xwlxdh (mgr.14182) 51 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:05.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:04 smithi088 bash[14069]: cluster 2024-01-13T11:49:03.864163+0000 mgr.smithi088.xwlxdh (mgr.14182) 51 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:05.735 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:49:07.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:06 smithi088 bash[14069]: cluster 2024-01-13T11:49:05.864667+0000 mgr.smithi088.xwlxdh (mgr.14182) 52 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:07.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:06 smithi138 bash[17620]: cluster 2024-01-13T11:49:05.864667+0000 mgr.smithi088.xwlxdh (mgr.14182) 52 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:09.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:08 smithi138 bash[17620]: cluster 2024-01-13T11:49:07.865518+0000 mgr.smithi088.xwlxdh (mgr.14182) 53 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:09.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:08 smithi138 bash[17620]: audit 2024-01-13T11:49:08.348429+0000 mon.smithi088 (mon.0) 236 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:49:09.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:08 smithi138 bash[17620]: audit 2024-01-13T11:49:08.349874+0000 mon.smithi088 (mon.0) 237 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi088", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-13T11:49:09.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:08 smithi138 bash[17620]: audit 2024-01-13T11:49:08.351278+0000 mon.smithi088 (mon.0) 238 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:49:09.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:08 smithi088 bash[14069]: cluster 2024-01-13T11:49:07.865518+0000 mgr.smithi088.xwlxdh (mgr.14182) 53 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:09.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:08 smithi088 bash[14069]: audit 2024-01-13T11:49:08.348429+0000 mon.smithi088 (mon.0) 236 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:49:09.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:08 smithi088 bash[14069]: audit 2024-01-13T11:49:08.349874+0000 mon.smithi088 (mon.0) 237 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi088", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-13T11:49:09.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:08 smithi088 bash[14069]: audit 2024-01-13T11:49:08.351278+0000 mon.smithi088 (mon.0) 238 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:49:10.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:09 smithi138 bash[17620]: cephadm 2024-01-13T11:49:08.349389+0000 mgr.smithi088.xwlxdh (mgr.14182) 54 : cephadm [INF] Reconfiguring crash.smithi088 (monmap changed)... 2024-01-13T11:49:10.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:09 smithi138 bash[17620]: cephadm 2024-01-13T11:49:08.352148+0000 mgr.smithi088.xwlxdh (mgr.14182) 55 : cephadm [INF] Reconfiguring daemon crash.smithi088 on smithi088 2024-01-13T11:49:10.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:09 smithi088 bash[14069]: cephadm 2024-01-13T11:49:08.349389+0000 mgr.smithi088.xwlxdh (mgr.14182) 54 : cephadm [INF] Reconfiguring crash.smithi088 (monmap changed)... 2024-01-13T11:49:10.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:09 smithi088 bash[14069]: cephadm 2024-01-13T11:49:08.352148+0000 mgr.smithi088.xwlxdh (mgr.14182) 55 : cephadm [INF] Reconfiguring daemon crash.smithi088 on smithi088 2024-01-13T11:49:10.645 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:49:10.892 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:10 smithi088 bash[14069]: cluster 2024-01-13T11:49:09.866530+0000 mgr.smithi088.xwlxdh (mgr.14182) 56 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:10.892 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:10 smithi088 bash[14069]: audit 2024-01-13T11:49:10.655688+0000 mon.smithi088 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.88:0/2773465821' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:49:11.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:10 smithi138 bash[17620]: cluster 2024-01-13T11:49:09.866530+0000 mgr.smithi088.xwlxdh (mgr.14182) 56 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:11.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:10 smithi138 bash[17620]: audit 2024-01-13T11:49:10.655688+0000 mon.smithi088 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.88:0/2773465821' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:49:11.990 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":9,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:49:12.549 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:12 smithi088 bash[14069]: audit 2024-01-13T11:49:11.365384+0000 mon.smithi088 (mon.0) 240 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:49:12.549 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:12 smithi088 bash[14069]: cephadm 2024-01-13T11:49:11.366283+0000 mgr.smithi088.xwlxdh (mgr.14182) 57 : cephadm [INF] Reconfiguring grafana.smithi088 (dependencies changed)... 2024-01-13T11:49:12.549 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:12 smithi088 bash[14069]: cephadm 2024-01-13T11:49:11.406984+0000 mgr.smithi088.xwlxdh (mgr.14182) 58 : cephadm [INF] Reconfiguring daemon grafana.smithi088 on smithi088 2024-01-13T11:49:12.549 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:12 smithi088 bash[14069]: cluster 2024-01-13T11:49:11.867539+0000 mgr.smithi088.xwlxdh (mgr.14182) 59 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:12.625 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:12 smithi138 bash[17620]: audit 2024-01-13T11:49:11.365384+0000 mon.smithi088 (mon.0) 240 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:49:12.625 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:12 smithi138 bash[17620]: cephadm 2024-01-13T11:49:11.366283+0000 mgr.smithi088.xwlxdh (mgr.14182) 57 : cephadm [INF] Reconfiguring grafana.smithi088 (dependencies changed)... 2024-01-13T11:49:12.625 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:12 smithi138 bash[17620]: cephadm 2024-01-13T11:49:11.406984+0000 mgr.smithi088.xwlxdh (mgr.14182) 58 : cephadm [INF] Reconfiguring daemon grafana.smithi088 on smithi088 2024-01-13T11:49:12.625 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:12 smithi138 bash[17620]: cluster 2024-01-13T11:49:11.867539+0000 mgr.smithi088.xwlxdh (mgr.14182) 59 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:12.991 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:49:15.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:14 smithi088 bash[14069]: cluster 2024-01-13T11:49:13.868574+0000 mgr.smithi088.xwlxdh (mgr.14182) 60 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:15.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:14 smithi138 bash[17620]: cluster 2024-01-13T11:49:13.868574+0000 mgr.smithi088.xwlxdh (mgr.14182) 60 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:17.096 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:16 smithi088 bash[14069]: cluster 2024-01-13T11:49:15.869511+0000 mgr.smithi088.xwlxdh (mgr.14182) 61 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:17.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:16 smithi138 bash[17620]: cluster 2024-01-13T11:49:15.869511+0000 mgr.smithi088.xwlxdh (mgr.14182) 61 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:18.104 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:49:19.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:18 smithi138 bash[17620]: audit 2024-01-13T11:49:17.841833+0000 mon.smithi088 (mon.0) 241 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:49:19.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:18 smithi138 bash[17620]: cephadm 2024-01-13T11:49:17.843050+0000 mgr.smithi088.xwlxdh (mgr.14182) 62 : cephadm [INF] Reconfiguring crash.smithi138 (monmap changed)... 2024-01-13T11:49:19.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:18 smithi138 bash[17620]: audit 2024-01-13T11:49:17.843417+0000 mon.smithi088 (mon.0) 242 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi138", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-13T11:49:19.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:18 smithi138 bash[17620]: audit 2024-01-13T11:49:17.844461+0000 mon.smithi088 (mon.0) 243 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:49:19.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:18 smithi138 bash[17620]: cephadm 2024-01-13T11:49:17.845245+0000 mgr.smithi088.xwlxdh (mgr.14182) 63 : cephadm [INF] Reconfiguring daemon crash.smithi138 on smithi138 2024-01-13T11:49:19.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:18 smithi138 bash[17620]: cluster 2024-01-13T11:49:17.869979+0000 mgr.smithi088.xwlxdh (mgr.14182) 64 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:19.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:18 smithi138 bash[17620]: audit 2024-01-13T11:49:18.114615+0000 mon.smithi088 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.88:0/3138906898' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:49:19.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:18 smithi088 bash[14069]: audit 2024-01-13T11:49:17.841833+0000 mon.smithi088 (mon.0) 241 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:49:19.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:18 smithi088 bash[14069]: cephadm 2024-01-13T11:49:17.843050+0000 mgr.smithi088.xwlxdh (mgr.14182) 62 : cephadm [INF] Reconfiguring crash.smithi138 (monmap changed)... 2024-01-13T11:49:19.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:18 smithi088 bash[14069]: audit 2024-01-13T11:49:17.843417+0000 mon.smithi088 (mon.0) 242 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi138", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-13T11:49:19.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:18 smithi088 bash[14069]: audit 2024-01-13T11:49:17.844461+0000 mon.smithi088 (mon.0) 243 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:49:19.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:18 smithi088 bash[14069]: cephadm 2024-01-13T11:49:17.845245+0000 mgr.smithi088.xwlxdh (mgr.14182) 63 : cephadm [INF] Reconfiguring daemon crash.smithi138 on smithi138 2024-01-13T11:49:19.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:18 smithi088 bash[14069]: cluster 2024-01-13T11:49:17.869979+0000 mgr.smithi088.xwlxdh (mgr.14182) 64 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:19.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:18 smithi088 bash[14069]: audit 2024-01-13T11:49:18.114615+0000 mon.smithi088 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.88:0/3138906898' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:49:19.234 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":9,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:49:20.235 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:49:21.065 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:20 smithi138 bash[17620]: audit 2024-01-13T11:49:19.806944+0000 mon.smithi088 (mon.0) 245 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:49:21.065 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:20 smithi138 bash[17620]: cephadm 2024-01-13T11:49:19.808387+0000 mgr.smithi088.xwlxdh (mgr.14182) 65 : cephadm [INF] Reconfiguring mgr.smithi138.divzmq (monmap changed)... 2024-01-13T11:49:21.065 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:20 smithi138 bash[17620]: audit 2024-01-13T11:49:19.808905+0000 mon.smithi088 (mon.0) 246 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi138.divzmq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-13T11:49:21.065 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:20 smithi138 bash[17620]: audit 2024-01-13T11:49:19.810542+0000 mon.smithi088 (mon.0) 247 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-13T11:49:21.065 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:20 smithi138 bash[17620]: audit 2024-01-13T11:49:19.811908+0000 mon.smithi088 (mon.0) 248 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:49:21.066 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:20 smithi138 bash[17620]: cephadm 2024-01-13T11:49:19.813135+0000 mgr.smithi088.xwlxdh (mgr.14182) 66 : cephadm [INF] Reconfiguring daemon mgr.smithi138.divzmq on smithi138 2024-01-13T11:49:21.066 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:20 smithi138 bash[17620]: cluster 2024-01-13T11:49:19.870994+0000 mgr.smithi088.xwlxdh (mgr.14182) 67 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:21.121 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:20 smithi088 bash[14069]: audit 2024-01-13T11:49:19.806944+0000 mon.smithi088 (mon.0) 245 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:49:21.122 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:20 smithi088 bash[14069]: cephadm 2024-01-13T11:49:19.808387+0000 mgr.smithi088.xwlxdh (mgr.14182) 65 : cephadm [INF] Reconfiguring mgr.smithi138.divzmq (monmap changed)... 2024-01-13T11:49:21.122 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:20 smithi088 bash[14069]: audit 2024-01-13T11:49:19.808905+0000 mon.smithi088 (mon.0) 246 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi138.divzmq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-13T11:49:21.122 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:20 smithi088 bash[14069]: audit 2024-01-13T11:49:19.810542+0000 mon.smithi088 (mon.0) 247 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-13T11:49:21.122 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:20 smithi088 bash[14069]: audit 2024-01-13T11:49:19.811908+0000 mon.smithi088 (mon.0) 248 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:49:21.122 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:20 smithi088 bash[14069]: cephadm 2024-01-13T11:49:19.813135+0000 mgr.smithi088.xwlxdh (mgr.14182) 66 : cephadm [INF] Reconfiguring daemon mgr.smithi138.divzmq on smithi138 2024-01-13T11:49:21.122 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:20 smithi088 bash[14069]: cluster 2024-01-13T11:49:19.870994+0000 mgr.smithi088.xwlxdh (mgr.14182) 67 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:22.795 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:22 smithi088 bash[14069]: audit 2024-01-13T11:49:21.521029+0000 mon.smithi088 (mon.0) 249 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:49:22.795 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:22 smithi088 bash[14069]: cephadm 2024-01-13T11:49:21.522050+0000 mgr.smithi088.xwlxdh (mgr.14182) 68 : cephadm [INF] Reconfiguring mon.smithi138 (monmap changed)... 2024-01-13T11:49:22.795 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:22 smithi088 bash[14069]: audit 2024-01-13T11:49:21.522504+0000 mon.smithi088 (mon.0) 250 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-13T11:49:22.796 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:22 smithi088 bash[14069]: audit 2024-01-13T11:49:21.523756+0000 mon.smithi088 (mon.0) 251 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-13T11:49:22.796 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:22 smithi088 bash[14069]: audit 2024-01-13T11:49:21.524826+0000 mon.smithi088 (mon.0) 252 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:49:22.796 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:22 smithi088 bash[14069]: cephadm 2024-01-13T11:49:21.525798+0000 mgr.smithi088.xwlxdh (mgr.14182) 69 : cephadm [INF] Reconfiguring daemon mon.smithi138 on smithi138 2024-01-13T11:49:22.796 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:22 smithi088 bash[14069]: cluster 2024-01-13T11:49:21.871554+0000 mgr.smithi088.xwlxdh (mgr.14182) 70 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:22.832 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:22 smithi138 bash[17620]: audit 2024-01-13T11:49:21.521029+0000 mon.smithi088 (mon.0) 249 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:49:22.832 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:22 smithi138 bash[17620]: cephadm 2024-01-13T11:49:21.522050+0000 mgr.smithi088.xwlxdh (mgr.14182) 68 : cephadm [INF] Reconfiguring mon.smithi138 (monmap changed)... 2024-01-13T11:49:22.832 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:22 smithi138 bash[17620]: audit 2024-01-13T11:49:21.522504+0000 mon.smithi088 (mon.0) 250 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-13T11:49:22.832 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:22 smithi138 bash[17620]: audit 2024-01-13T11:49:21.523756+0000 mon.smithi088 (mon.0) 251 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-13T11:49:22.832 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:22 smithi138 bash[17620]: audit 2024-01-13T11:49:21.524826+0000 mon.smithi088 (mon.0) 252 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:49:22.833 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:22 smithi138 bash[17620]: cephadm 2024-01-13T11:49:21.525798+0000 mgr.smithi088.xwlxdh (mgr.14182) 69 : cephadm [INF] Reconfiguring daemon mon.smithi138 on smithi138 2024-01-13T11:49:22.833 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:22 smithi138 bash[17620]: cluster 2024-01-13T11:49:21.871554+0000 mgr.smithi088.xwlxdh (mgr.14182) 70 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:23.667 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:49:24.402 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":9,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:49:24.625 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:24 smithi138 bash[17620]: audit 2024-01-13T11:49:23.310694+0000 mon.smithi088 (mon.0) 253 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:49:24.625 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:24 smithi138 bash[17620]: audit 2024-01-13T11:49:23.315529+0000 mon.smithi088 (mon.0) 254 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-13T11:49:24.625 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:24 smithi138 bash[17620]: audit 2024-01-13T11:49:23.316117+0000 mgr.smithi088.xwlxdh (mgr.14182) 71 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-13T11:49:24.626 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:24 smithi138 bash[17620]: audit 2024-01-13T11:49:23.317693+0000 mon.smithi088 (mon.0) 255 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi088.front.sepia.ceph.com:9093"}]: dispatch 2024-01-13T11:49:24.626 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:24 smithi138 bash[17620]: audit 2024-01-13T11:49:23.318355+0000 mgr.smithi088.xwlxdh (mgr.14182) 72 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi088.front.sepia.ceph.com:9093"}]: dispatch 2024-01-13T11:49:24.626 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:24 smithi138 bash[17620]: audit 2024-01-13T11:49:23.323806+0000 mon.smithi088 (mon.0) 256 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:49:24.626 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:24 smithi138 bash[17620]: audit 2024-01-13T11:49:23.333579+0000 mon.smithi088 (mon.0) 257 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-13T11:49:24.626 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:24 smithi138 bash[17620]: audit 2024-01-13T11:49:23.334307+0000 mgr.smithi088.xwlxdh (mgr.14182) 73 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-13T11:49:24.626 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:24 smithi138 bash[17620]: audit 2024-01-13T11:49:23.342262+0000 mon.smithi088 (mon.0) 258 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi088.front.sepia.ceph.com:3000"}]: dispatch 2024-01-13T11:49:24.626 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:24 smithi138 bash[17620]: audit 2024-01-13T11:49:23.342971+0000 mgr.smithi088.xwlxdh (mgr.14182) 74 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi088.front.sepia.ceph.com:3000"}]: dispatch 2024-01-13T11:49:24.626 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:24 smithi138 bash[17620]: audit 2024-01-13T11:49:23.350439+0000 mon.smithi088 (mon.0) 259 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:49:24.626 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:24 smithi138 bash[17620]: audit 2024-01-13T11:49:23.362244+0000 mon.smithi088 (mon.0) 260 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-01-13T11:49:24.626 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:24 smithi138 bash[17620]: audit 2024-01-13T11:49:23.362989+0000 mgr.smithi088.xwlxdh (mgr.14182) 75 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-01-13T11:49:24.626 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:24 smithi138 bash[17620]: audit 2024-01-13T11:49:23.366217+0000 mon.smithi088 (mon.0) 261 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi138.front.sepia.ceph.com:9095"}]: dispatch 2024-01-13T11:49:24.626 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:24 smithi138 bash[17620]: audit 2024-01-13T11:49:23.366720+0000 mgr.smithi088.xwlxdh (mgr.14182) 76 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi138.front.sepia.ceph.com:9095"}]: dispatch 2024-01-13T11:49:24.626 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:24 smithi138 bash[17620]: audit 2024-01-13T11:49:23.372948+0000 mon.smithi088 (mon.0) 262 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:49:24.627 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:24 smithi138 bash[17620]: audit 2024-01-13T11:49:23.376170+0000 mon.smithi088 (mon.0) 263 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:49:24.627 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:24 smithi138 bash[17620]: audit 2024-01-13T11:49:23.677455+0000 mon.smithi088 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.88:0/3591398771' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:49:24.627 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:24 smithi138 bash[17620]: cluster 2024-01-13T11:49:23.872422+0000 mgr.smithi088.xwlxdh (mgr.14182) 77 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:24.642 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:24 smithi088 bash[14069]: audit 2024-01-13T11:49:23.310694+0000 mon.smithi088 (mon.0) 253 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:49:24.642 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:24 smithi088 bash[14069]: audit 2024-01-13T11:49:23.315529+0000 mon.smithi088 (mon.0) 254 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-13T11:49:24.642 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:24 smithi088 bash[14069]: audit 2024-01-13T11:49:23.316117+0000 mgr.smithi088.xwlxdh (mgr.14182) 71 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-13T11:49:24.643 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:24 smithi088 bash[14069]: audit 2024-01-13T11:49:23.317693+0000 mon.smithi088 (mon.0) 255 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi088.front.sepia.ceph.com:9093"}]: dispatch 2024-01-13T11:49:24.643 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:24 smithi088 bash[14069]: audit 2024-01-13T11:49:23.318355+0000 mgr.smithi088.xwlxdh (mgr.14182) 72 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi088.front.sepia.ceph.com:9093"}]: dispatch 2024-01-13T11:49:24.643 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:24 smithi088 bash[14069]: audit 2024-01-13T11:49:23.323806+0000 mon.smithi088 (mon.0) 256 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:49:24.643 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:24 smithi088 bash[14069]: audit 2024-01-13T11:49:23.333579+0000 mon.smithi088 (mon.0) 257 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-13T11:49:24.643 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:24 smithi088 bash[14069]: audit 2024-01-13T11:49:23.334307+0000 mgr.smithi088.xwlxdh (mgr.14182) 73 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-13T11:49:24.643 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:24 smithi088 bash[14069]: audit 2024-01-13T11:49:23.342262+0000 mon.smithi088 (mon.0) 258 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi088.front.sepia.ceph.com:3000"}]: dispatch 2024-01-13T11:49:24.643 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:24 smithi088 bash[14069]: audit 2024-01-13T11:49:23.342971+0000 mgr.smithi088.xwlxdh (mgr.14182) 74 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi088.front.sepia.ceph.com:3000"}]: dispatch 2024-01-13T11:49:24.643 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:24 smithi088 bash[14069]: audit 2024-01-13T11:49:23.350439+0000 mon.smithi088 (mon.0) 259 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:49:24.643 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:24 smithi088 bash[14069]: audit 2024-01-13T11:49:23.362244+0000 mon.smithi088 (mon.0) 260 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-01-13T11:49:24.643 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:24 smithi088 bash[14069]: audit 2024-01-13T11:49:23.362989+0000 mgr.smithi088.xwlxdh (mgr.14182) 75 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-01-13T11:49:24.643 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:24 smithi088 bash[14069]: audit 2024-01-13T11:49:23.366217+0000 mon.smithi088 (mon.0) 261 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi138.front.sepia.ceph.com:9095"}]: dispatch 2024-01-13T11:49:24.643 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:24 smithi088 bash[14069]: audit 2024-01-13T11:49:23.366720+0000 mgr.smithi088.xwlxdh (mgr.14182) 76 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi138.front.sepia.ceph.com:9095"}]: dispatch 2024-01-13T11:49:24.643 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:24 smithi088 bash[14069]: audit 2024-01-13T11:49:23.372948+0000 mon.smithi088 (mon.0) 262 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:49:24.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:24 smithi088 bash[14069]: audit 2024-01-13T11:49:23.376170+0000 mon.smithi088 (mon.0) 263 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:49:24.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:24 smithi088 bash[14069]: audit 2024-01-13T11:49:23.677455+0000 mon.smithi088 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.88:0/3591398771' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:49:24.644 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:24 smithi088 bash[14069]: cluster 2024-01-13T11:49:23.872422+0000 mgr.smithi088.xwlxdh (mgr.14182) 77 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:25.403 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:49:27.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:26 smithi088 bash[14069]: cluster 2024-01-13T11:49:25.873510+0000 mgr.smithi088.xwlxdh (mgr.14182) 78 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:27.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:26 smithi138 bash[17620]: cluster 2024-01-13T11:49:25.873510+0000 mgr.smithi088.xwlxdh (mgr.14182) 78 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:28.452 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:49:29.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:28 smithi138 bash[17620]: cluster 2024-01-13T11:49:27.874533+0000 mgr.smithi088.xwlxdh (mgr.14182) 79 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:29.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:28 smithi138 bash[17620]: audit 2024-01-13T11:49:28.462771+0000 mon.smithi088 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.88:0/2723070912' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:49:29.125 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:28 smithi088 bash[14069]: cluster 2024-01-13T11:49:27.874533+0000 mgr.smithi088.xwlxdh (mgr.14182) 79 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:29.125 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:28 smithi088 bash[14069]: audit 2024-01-13T11:49:28.462771+0000 mon.smithi088 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.88:0/2723070912' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:49:29.126 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":9,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:49:30.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:29 smithi138 bash[17620]: audit 2024-01-13T11:49:28.860383+0000 mon.smithi088 (mon.0) 266 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:49:30.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:29 smithi138 bash[17620]: audit 2024-01-13T11:49:28.869105+0000 mon.smithi088 (mon.0) 267 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:49:30.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:29 smithi138 bash[17620]: audit 2024-01-13T11:49:29.138947+0000 mon.smithi088 (mon.0) 268 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:49:30.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:29 smithi138 bash[17620]: audit 2024-01-13T11:49:29.140477+0000 mon.smithi088 (mon.0) 269 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:49:30.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:29 smithi138 bash[17620]: audit 2024-01-13T11:49:29.141580+0000 mon.smithi088 (mon.0) 270 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T11:49:30.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:29 smithi138 bash[17620]: audit 2024-01-13T11:49:29.147577+0000 mon.smithi088 (mon.0) 271 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:49:30.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:29 smithi138 bash[17620]: audit 2024-01-13T11:49:29.153645+0000 mon.smithi088 (mon.0) 272 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T11:49:30.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:29 smithi138 bash[17620]: audit 2024-01-13T11:49:29.159746+0000 mon.smithi088 (mon.0) 273 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:49:30.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:29 smithi138 bash[17620]: audit 2024-01-13T11:49:29.161679+0000 mon.smithi088 (mon.0) 274 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:49:30.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:29 smithi138 bash[17620]: audit 2024-01-13T11:49:29.162071+0000 mon.smithi088 (mon.0) 275 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:49:30.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:29 smithi138 bash[17620]: audit 2024-01-13T11:49:29.162884+0000 mon.smithi088 (mon.0) 276 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:49:30.127 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:49:30.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:29 smithi088 bash[14069]: audit 2024-01-13T11:49:28.860383+0000 mon.smithi088 (mon.0) 266 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:49:30.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:29 smithi088 bash[14069]: audit 2024-01-13T11:49:28.869105+0000 mon.smithi088 (mon.0) 267 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:49:30.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:29 smithi088 bash[14069]: audit 2024-01-13T11:49:29.138947+0000 mon.smithi088 (mon.0) 268 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:49:30.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:29 smithi088 bash[14069]: audit 2024-01-13T11:49:29.140477+0000 mon.smithi088 (mon.0) 269 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:49:30.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:29 smithi088 bash[14069]: audit 2024-01-13T11:49:29.141580+0000 mon.smithi088 (mon.0) 270 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T11:49:30.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:29 smithi088 bash[14069]: audit 2024-01-13T11:49:29.147577+0000 mon.smithi088 (mon.0) 271 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:49:30.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:29 smithi088 bash[14069]: audit 2024-01-13T11:49:29.153645+0000 mon.smithi088 (mon.0) 272 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T11:49:30.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:29 smithi088 bash[14069]: audit 2024-01-13T11:49:29.159746+0000 mon.smithi088 (mon.0) 273 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:49:30.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:29 smithi088 bash[14069]: audit 2024-01-13T11:49:29.161679+0000 mon.smithi088 (mon.0) 274 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:49:30.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:29 smithi088 bash[14069]: audit 2024-01-13T11:49:29.162071+0000 mon.smithi088 (mon.0) 275 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:49:30.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:29 smithi088 bash[14069]: audit 2024-01-13T11:49:29.162884+0000 mon.smithi088 (mon.0) 276 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:49:31.122 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:30 smithi088 bash[14069]: cluster 2024-01-13T11:49:29.148100+0000 mgr.smithi088.xwlxdh (mgr.14182) 80 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:31.122 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:30 smithi088 bash[14069]: cluster 2024-01-13T11:49:30.145608+0000 mon.smithi088 (mon.0) 277 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T11:49:31.122 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:30 smithi088 bash[14069]: cluster 2024-01-13T11:49:30.145665+0000 mon.smithi088 (mon.0) 278 : cluster [INF] Cluster is now healthy 2024-01-13T11:49:31.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:30 smithi138 bash[17620]: cluster 2024-01-13T11:49:29.148100+0000 mgr.smithi088.xwlxdh (mgr.14182) 80 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:31.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:30 smithi138 bash[17620]: cluster 2024-01-13T11:49:30.145608+0000 mon.smithi088 (mon.0) 277 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T11:49:31.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:30 smithi138 bash[17620]: cluster 2024-01-13T11:49:30.145665+0000 mon.smithi088 (mon.0) 278 : cluster [INF] Cluster is now healthy 2024-01-13T11:49:33.121 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:49:33.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:32 smithi138 bash[17620]: cluster 2024-01-13T11:49:31.148508+0000 mgr.smithi088.xwlxdh (mgr.14182) 81 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:33.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:32 smithi088 bash[14069]: cluster 2024-01-13T11:49:31.148508+0000 mgr.smithi088.xwlxdh (mgr.14182) 81 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:33.815 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":9,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:49:34.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:33 smithi138 bash[17620]: audit 2024-01-13T11:49:33.131158+0000 mon.smithi088 (mon.0) 279 : audit [DBG] from='client.? 172.21.15.88:0/706620184' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:49:34.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:33 smithi088 bash[14069]: audit 2024-01-13T11:49:33.131158+0000 mon.smithi088 (mon.0) 279 : audit [DBG] from='client.? 172.21.15.88:0/706620184' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:49:34.817 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:49:35.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:34 smithi138 bash[17620]: cluster 2024-01-13T11:49:33.148903+0000 mgr.smithi088.xwlxdh (mgr.14182) 82 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:35.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:34 smithi088 bash[14069]: cluster 2024-01-13T11:49:33.148903+0000 mgr.smithi088.xwlxdh (mgr.14182) 82 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:37.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:36 smithi088 bash[14069]: cluster 2024-01-13T11:49:35.149313+0000 mgr.smithi088.xwlxdh (mgr.14182) 83 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:37.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:36 smithi138 bash[17620]: cluster 2024-01-13T11:49:35.149313+0000 mgr.smithi088.xwlxdh (mgr.14182) 83 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:39.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:38 smithi138 bash[17620]: cluster 2024-01-13T11:49:37.149709+0000 mgr.smithi088.xwlxdh (mgr.14182) 84 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:39.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:38 smithi138 bash[17620]: audit 2024-01-13T11:49:38.024405+0000 mon.smithi138 (mon.1) 4 : audit [INF] from='client.? 172.21.15.138:0/287866790' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb8b8805-c3d6-4854-9aa3-b6bc501e43c8"}]: dispatch 2024-01-13T11:49:39.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:38 smithi138 bash[17620]: audit 2024-01-13T11:49:38.025451+0000 mon.smithi088 (mon.0) 280 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb8b8805-c3d6-4854-9aa3-b6bc501e43c8"}]: dispatch 2024-01-13T11:49:39.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:38 smithi138 bash[17620]: audit 2024-01-13T11:49:38.032193+0000 mon.smithi088 (mon.0) 281 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb8b8805-c3d6-4854-9aa3-b6bc501e43c8"}]': finished 2024-01-13T11:49:39.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:38 smithi138 bash[17620]: cluster 2024-01-13T11:49:38.032293+0000 mon.smithi088 (mon.0) 282 : cluster [DBG] osdmap e10: 1 total, 0 up, 1 in 2024-01-13T11:49:39.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:38 smithi138 bash[17620]: audit 2024-01-13T11:49:38.032486+0000 mon.smithi088 (mon.0) 283 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:49:39.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:38 smithi138 bash[17620]: audit 2024-01-13T11:49:38.378787+0000 mon.smithi138 (mon.1) 5 : audit [INF] from='client.? 172.21.15.138:0/2092626954' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:49:39.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:38 smithi138 bash[17620]: audit 2024-01-13T11:49:38.379670+0000 mon.smithi088 (mon.0) 284 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:49:39.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:38 smithi138 bash[17620]: audit 2024-01-13T11:49:38.388629+0000 mon.smithi088 (mon.0) 285 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:49:39.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:38 smithi138 bash[17620]: cluster 2024-01-13T11:49:38.388699+0000 mon.smithi088 (mon.0) 286 : cluster [DBG] osdmap e11: 0 total, 0 up, 0 in 2024-01-13T11:49:39.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:38 smithi088 bash[14069]: cluster 2024-01-13T11:49:37.149709+0000 mgr.smithi088.xwlxdh (mgr.14182) 84 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:39.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:38 smithi088 bash[14069]: audit 2024-01-13T11:49:38.024405+0000 mon.smithi138 (mon.1) 4 : audit [INF] from='client.? 172.21.15.138:0/287866790' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb8b8805-c3d6-4854-9aa3-b6bc501e43c8"}]: dispatch 2024-01-13T11:49:39.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:38 smithi088 bash[14069]: audit 2024-01-13T11:49:38.025451+0000 mon.smithi088 (mon.0) 280 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb8b8805-c3d6-4854-9aa3-b6bc501e43c8"}]: dispatch 2024-01-13T11:49:39.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:38 smithi088 bash[14069]: audit 2024-01-13T11:49:38.032193+0000 mon.smithi088 (mon.0) 281 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb8b8805-c3d6-4854-9aa3-b6bc501e43c8"}]': finished 2024-01-13T11:49:39.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:38 smithi088 bash[14069]: cluster 2024-01-13T11:49:38.032293+0000 mon.smithi088 (mon.0) 282 : cluster [DBG] osdmap e10: 1 total, 0 up, 1 in 2024-01-13T11:49:39.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:38 smithi088 bash[14069]: audit 2024-01-13T11:49:38.032486+0000 mon.smithi088 (mon.0) 283 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:49:39.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:38 smithi088 bash[14069]: audit 2024-01-13T11:49:38.378787+0000 mon.smithi138 (mon.1) 5 : audit [INF] from='client.? 172.21.15.138:0/2092626954' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:49:39.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:38 smithi088 bash[14069]: audit 2024-01-13T11:49:38.379670+0000 mon.smithi088 (mon.0) 284 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:49:39.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:38 smithi088 bash[14069]: audit 2024-01-13T11:49:38.388629+0000 mon.smithi088 (mon.0) 285 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:49:39.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:38 smithi088 bash[14069]: cluster 2024-01-13T11:49:38.388699+0000 mon.smithi088 (mon.0) 286 : cluster [DBG] osdmap e11: 0 total, 0 up, 0 in 2024-01-13T11:49:39.329 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:49:40.095 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":11,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:49:40.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:39 smithi138 bash[17620]: audit 2024-01-13T11:49:39.339263+0000 mon.smithi088 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.88:0/3848012022' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:49:40.392 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:39 smithi088 bash[14069]: audit 2024-01-13T11:49:39.339263+0000 mon.smithi088 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.88:0/3848012022' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:49:41.096 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:49:41.221 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: cluster 2024-01-13T11:49:39.149978+0000 mgr.smithi088.xwlxdh (mgr.14182) 85 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:41.222 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: cephadm 2024-01-13T11:49:39.255897+0000 mgr.smithi088.xwlxdh (mgr.14182) 86 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:49:41.222 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:49:41.222 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:49:41.222 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: return f(*arg) 2024-01-13T11:49:41.222 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:49:41.222 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:49:41.222 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:49:41.222 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: code, '\n'.join(err))) 2024-01-13T11:49:41.222 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:49:41.222 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmphoev5vri:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp0e8hz73y:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:49:41.222 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:49:41.222 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:49:41.222 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:49:41.223 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new eb8b8805-c3d6-4854-9aa3-b6bc501e43c8 2024-01-13T11:49:41.223 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:49:41.223 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:49:41.223 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:49:41.223 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:49:41.223 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:49:41.223 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:49:41.224 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:49:41.224 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:49:41.224 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:49:41.224 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:49:41.224 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:49:41.224 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:49:41.224 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:49:41.224 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:49:41.224 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:49:41.224 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:49:41.225 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:49:41.225 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:49:41.225 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:49:41.225 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:49:41.225 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:49:41.225 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:49:41.225 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:49:41.225 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:49:41.225 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:49:41.225 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:49:41.225 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:49:41.225 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:49:41.225 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:49:41.225 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:49:41.226 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:49:41.226 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:49:41.226 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:49:41.226 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:49:41.226 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:49:41.226 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:49:41.226 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:49:41.226 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:49:41.226 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:49:41.226 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:49:41.226 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:49:41.226 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:49:41.226 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:49:41.226 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:49:41.227 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:49:41.227 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:49:41.227 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:49:41.227 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:49:41.227 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:49:41.227 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:49:41.227 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:49:41.227 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:49:41.227 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:49:41.227 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:49:41.227 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:49:41.227 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:49:41.227 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:49:41.227 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:49:41.228 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: main() 2024-01-13T11:49:41.228 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:49:41.228 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:49:41.228 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:49:41.228 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:49:41.228 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:49:41.228 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:49:41.228 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:49:41.228 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:49:41.228 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:49:41.228 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:49:41.229 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:49:41.229 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:49:41.229 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:49:41.229 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:49:41.229 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmphoev5vri:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp0e8hz73y:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:49:41.229 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: audit 2024-01-13T11:49:40.682120+0000 mon.smithi088 (mon.0) 288 : audit [INF] from='client.? 172.21.15.88:0/1834310366' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "abfd5318-f296-40eb-bf6e-6fe45d6aed6b"}]: dispatch 2024-01-13T11:49:41.229 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: audit 2024-01-13T11:49:40.691116+0000 mon.smithi088 (mon.0) 289 : audit [INF] from='client.? 172.21.15.88:0/1834310366' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "abfd5318-f296-40eb-bf6e-6fe45d6aed6b"}]': finished 2024-01-13T11:49:41.229 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: cluster 2024-01-13T11:49:40.691202+0000 mon.smithi088 (mon.0) 290 : cluster [DBG] osdmap e12: 1 total, 0 up, 1 in 2024-01-13T11:49:41.229 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:40 smithi088 bash[14069]: audit 2024-01-13T11:49:40.691353+0000 mon.smithi088 (mon.0) 291 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:49:41.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: cluster 2024-01-13T11:49:39.149978+0000 mgr.smithi088.xwlxdh (mgr.14182) 85 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:41.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: cephadm 2024-01-13T11:49:39.255897+0000 mgr.smithi088.xwlxdh (mgr.14182) 86 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:49:41.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:49:41.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:49:41.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: return f(*arg) 2024-01-13T11:49:41.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:49:41.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:49:41.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:49:41.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: code, '\n'.join(err))) 2024-01-13T11:49:41.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:49:41.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmphoev5vri:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp0e8hz73y:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:49:41.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:49:41.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:49:41.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:49:41.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new eb8b8805-c3d6-4854-9aa3-b6bc501e43c8 2024-01-13T11:49:41.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:49:41.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:49:41.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:49:41.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:49:41.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:49:41.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:49:41.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:49:41.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:49:41.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:49:41.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:49:41.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:49:41.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:49:41.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:49:41.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:49:41.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:49:41.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:49:41.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:49:41.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:49:41.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:49:41.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:49:41.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:49:41.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:49:41.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:49:41.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:49:41.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:49:41.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:49:41.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:49:41.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:49:41.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:49:41.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:49:41.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:49:41.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:49:41.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:49:41.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:49:41.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:49:41.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:49:41.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:49:41.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:49:41.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:49:41.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:49:41.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:49:41.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:49:41.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:49:41.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:49:41.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:49:41.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:49:41.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:49:41.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:49:41.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:49:41.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:49:41.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:49:41.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:49:41.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:49:41.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:49:41.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:49:41.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:49:41.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:49:41.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:49:41.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: main() 2024-01-13T11:49:41.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:49:41.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:49:41.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:49:41.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:49:41.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:49:41.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:49:41.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:49:41.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:49:41.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:49:41.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:49:41.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:49:41.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:49:41.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:49:41.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:49:41.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmphoev5vri:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp0e8hz73y:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:49:41.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: audit 2024-01-13T11:49:40.682120+0000 mon.smithi088 (mon.0) 288 : audit [INF] from='client.? 172.21.15.88:0/1834310366' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "abfd5318-f296-40eb-bf6e-6fe45d6aed6b"}]: dispatch 2024-01-13T11:49:41.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: audit 2024-01-13T11:49:40.691116+0000 mon.smithi088 (mon.0) 289 : audit [INF] from='client.? 172.21.15.88:0/1834310366' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "abfd5318-f296-40eb-bf6e-6fe45d6aed6b"}]': finished 2024-01-13T11:49:41.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: cluster 2024-01-13T11:49:40.691202+0000 mon.smithi088 (mon.0) 290 : cluster [DBG] osdmap e12: 1 total, 0 up, 1 in 2024-01-13T11:49:41.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:40 smithi138 bash[17620]: audit 2024-01-13T11:49:40.691353+0000 mon.smithi088 (mon.0) 291 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:49:42.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:41 smithi088 bash[14069]: audit 2024-01-13T11:49:41.039281+0000 mon.smithi088 (mon.0) 292 : audit [INF] from='client.? 172.21.15.88:0/3341153973' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:49:42.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:41 smithi088 bash[14069]: audit 2024-01-13T11:49:41.046370+0000 mon.smithi088 (mon.0) 293 : audit [INF] from='client.? 172.21.15.88:0/3341153973' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:49:42.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:41 smithi088 bash[14069]: cluster 2024-01-13T11:49:41.046461+0000 mon.smithi088 (mon.0) 294 : cluster [DBG] osdmap e13: 0 total, 0 up, 0 in 2024-01-13T11:49:42.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:41 smithi138 bash[17620]: audit 2024-01-13T11:49:41.039281+0000 mon.smithi088 (mon.0) 292 : audit [INF] from='client.? 172.21.15.88:0/3341153973' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:49:42.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:41 smithi138 bash[17620]: audit 2024-01-13T11:49:41.046370+0000 mon.smithi088 (mon.0) 293 : audit [INF] from='client.? 172.21.15.88:0/3341153973' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:49:42.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:41 smithi138 bash[17620]: cluster 2024-01-13T11:49:41.046461+0000 mon.smithi088 (mon.0) 294 : cluster [DBG] osdmap e13: 0 total, 0 up, 0 in 2024-01-13T11:49:43.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:42 smithi138 bash[17620]: cluster 2024-01-13T11:49:41.150417+0000 mgr.smithi088.xwlxdh (mgr.14182) 87 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:43.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:42 smithi138 bash[17620]: audit 2024-01-13T11:49:42.551225+0000 mon.smithi088 (mon.0) 295 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:49:43.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:42 smithi088 bash[14069]: cluster 2024-01-13T11:49:41.150417+0000 mgr.smithi088.xwlxdh (mgr.14182) 87 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:43.392 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:42 smithi088 bash[14069]: audit 2024-01-13T11:49:42.551225+0000 mon.smithi088 (mon.0) 295 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:49:44.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: cephadm 2024-01-13T11:49:42.543097+0000 mgr.smithi088.xwlxdh (mgr.14182) 88 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:49:44.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:49:44.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:49:44.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: return f(*arg) 2024-01-13T11:49:44.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:49:44.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:49:44.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:49:44.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: code, '\n'.join(err))) 2024-01-13T11:49:44.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi088/config 2024-01-13T11:49:44.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmprtvwjhf0:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp_hhj55wy:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:49:44.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:49:44.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:49:44.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:49:44.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new abfd5318-f296-40eb-bf6e-6fe45d6aed6b 2024-01-13T11:49:44.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:49:44.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:49:44.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:49:44.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:49:44.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:49:44.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:49:44.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:49:44.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:49:44.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:49:44.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:49:44.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:49:44.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:49:44.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:49:44.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:49:44.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:49:44.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:49:44.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:49:44.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:49:44.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:49:44.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:49:44.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:49:44.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:49:44.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:49:44.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:49:44.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:49:44.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:49:44.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:49:44.147 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:49:44.147 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:49:44.147 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:49:44.147 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:49:44.147 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:49:44.147 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:49:44.147 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:49:44.147 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:49:44.147 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:49:44.147 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:49:44.147 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:49:44.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:49:44.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:49:44.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:49:44.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:49:44.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:49:44.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:49:44.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:49:44.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:49:44.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:49:44.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:49:44.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:49:44.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:49:44.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:49:44.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:49:44.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:49:44.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:49:44.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:49:44.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:49:44.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:49:44.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:49:44.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: main() 2024-01-13T11:49:44.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:49:44.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:49:44.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:49:44.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:49:44.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:49:44.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:49:44.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:49:44.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:49:44.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:49:44.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:49:44.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:49:44.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:49:44.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:49:44.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:49:44.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmprtvwjhf0:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp_hhj55wy:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:49:44.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: cephadm 2024-01-13T11:49:42.547476+0000 mgr.smithi088.xwlxdh (mgr.14182) 89 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T11:49:44.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: service_id: all-available-devices 2024-01-13T11:49:44.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: service_name: osd.all-available-devices 2024-01-13T11:49:44.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: placement: 2024-01-13T11:49:44.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: host_pattern: '*' 2024-01-13T11:49:44.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: spec: 2024-01-13T11:49:44.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: data_devices: 2024-01-13T11:49:44.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: all: true 2024-01-13T11:49:44.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: filter_logic: AND 2024-01-13T11:49:44.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: objectstore: bluestore 2024-01-13T11:49:44.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:49:44.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmphoev5vri:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp0e8hz73y:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:49:44.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:49:44.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:49:44.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:49:44.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new eb8b8805-c3d6-4854-9aa3-b6bc501e43c8 2024-01-13T11:49:44.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:49:44.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:49:44.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:49:44.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:49:44.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:49:44.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:49:44.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:49:44.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:49:44.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:49:44.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:49:44.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:49:44.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:49:44.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:49:44.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:49:44.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:49:44.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:49:44.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:49:44.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:49:44.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:49:44.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:49:44.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:49:44.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:49:44.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:49:44.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:49:44.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:49:44.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:49:44.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:49:44.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:49:44.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:49:44.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:49:44.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:49:44.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:49:44.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:49:44.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:49:44.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:49:44.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:49:44.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:49:44.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:49:44.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:49:44.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:49:44.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:49:44.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:49:44.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:49:44.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:49:44.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:49:44.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:49:44.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:49:44.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:49:44.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:49:44.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:49:44.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:49:44.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:49:44.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:49:44.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:49:44.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:49:44.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:49:44.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:49:44.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:49:44.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: main() 2024-01-13T11:49:44.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:49:44.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:49:44.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:49:44.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:49:44.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:49:44.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:49:44.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:49:44.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:49:44.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:49:44.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:49:44.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:49:44.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:49:44.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:49:44.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:49:44.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmphoev5vri:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp0e8hz73y:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:49:44.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:49:44.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T11:49:44.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: if self._apply_service(spec): 2024-01-13T11:49:44.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T11:49:44.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T11:49:44.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T11:49:44.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T11:49:44.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T11:49:44.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T11:49:44.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T11:49:44.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T11:49:44.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T11:49:44.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: raise self._value 2024-01-13T11:49:44.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T11:49:44.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: result = (True, func(*args, **kwds)) 2024-01-13T11:49:44.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T11:49:44.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: return list(map(*args)) 2024-01-13T11:49:44.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:49:44.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: return f(*arg) 2024-01-13T11:49:44.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:49:44.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:49:44.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:49:44.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: code, '\n'.join(err))) 2024-01-13T11:49:44.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:49:44.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmphoev5vri:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp0e8hz73y:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:49:44.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:49:44.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:49:44.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:49:44.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new eb8b8805-c3d6-4854-9aa3-b6bc501e43c8 2024-01-13T11:49:44.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:49:44.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:49:44.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:49:44.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:49:44.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:49:44.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:49:44.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:49:44.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:49:44.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:49:44.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:49:44.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:49:44.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:49:44.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:49:44.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:49:44.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:49:44.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:49:44.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:49:44.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:49:44.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:49:44.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:49:44.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:49:44.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:49:44.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:49:44.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:49:44.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:49:44.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:49:44.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:49:44.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:49:44.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:49:44.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:49:44.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:49:44.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:49:44.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:49:44.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:49:44.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:49:44.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:49:44.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:49:44.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:49:44.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:49:44.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:49:44.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:49:44.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:49:44.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:49:44.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:49:44.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:49:44.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:49:44.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:49:44.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:49:44.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:49:44.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:49:44.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:49:44.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:49:44.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:49:44.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:49:44.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:49:44.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:49:44.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:49:44.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:49:44.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: main() 2024-01-13T11:49:44.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:49:44.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:49:44.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:49:44.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:49:44.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:49:44.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:49:44.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:49:44.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:49:44.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:49:44.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:49:44.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:49:44.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:49:44.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:49:44.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:49:44.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmphoev5vri:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp0e8hz73y:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:49:44.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: cluster 2024-01-13T11:49:42.548645+0000 mgr.smithi088.xwlxdh (mgr.14182) 90 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:44.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:43 smithi088 bash[14069]: cluster 2024-01-13T11:49:43.045486+0000 mon.smithi088 (mon.0) 296 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T11:49:44.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: cephadm 2024-01-13T11:49:42.543097+0000 mgr.smithi088.xwlxdh (mgr.14182) 88 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:49:44.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:49:44.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:49:44.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: return f(*arg) 2024-01-13T11:49:44.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:49:44.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:49:44.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:49:44.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: code, '\n'.join(err))) 2024-01-13T11:49:44.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi088/config 2024-01-13T11:49:44.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmprtvwjhf0:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp_hhj55wy:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:49:44.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:49:44.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:49:44.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:49:44.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new abfd5318-f296-40eb-bf6e-6fe45d6aed6b 2024-01-13T11:49:44.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:49:44.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:49:44.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:49:44.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:49:44.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:49:44.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:49:44.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:49:44.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:49:44.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:49:44.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:49:44.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:49:44.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:49:44.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:49:44.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:49:44.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:49:44.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:49:44.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:49:44.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:49:44.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:49:44.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:49:44.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:49:44.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:49:44.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:49:44.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:49:44.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:49:44.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:49:44.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:49:44.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:49:44.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:49:44.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:49:44.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:49:44.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:49:44.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:49:44.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:49:44.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:49:44.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:49:44.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:49:44.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:49:44.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:49:44.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:49:44.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:49:44.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:49:44.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:49:44.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:49:44.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:49:44.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:49:44.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:49:44.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:49:44.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:49:44.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:49:44.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:49:44.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:49:44.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:49:44.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:49:44.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:49:44.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:49:44.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:49:44.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:49:44.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: main() 2024-01-13T11:49:44.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:49:44.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:49:44.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:49:44.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:49:44.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:49:44.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:49:44.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:49:44.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:49:44.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:49:44.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:49:44.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:49:44.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:49:44.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:49:44.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:49:44.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmprtvwjhf0:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp_hhj55wy:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:49:44.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: cephadm 2024-01-13T11:49:42.547476+0000 mgr.smithi088.xwlxdh (mgr.14182) 89 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T11:49:44.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: service_id: all-available-devices 2024-01-13T11:49:44.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: service_name: osd.all-available-devices 2024-01-13T11:49:44.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: placement: 2024-01-13T11:49:44.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: host_pattern: '*' 2024-01-13T11:49:44.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: spec: 2024-01-13T11:49:44.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: data_devices: 2024-01-13T11:49:44.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: all: true 2024-01-13T11:49:44.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: filter_logic: AND 2024-01-13T11:49:44.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: objectstore: bluestore 2024-01-13T11:49:44.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:49:44.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmphoev5vri:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp0e8hz73y:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:49:44.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:49:44.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:49:44.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:49:44.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new eb8b8805-c3d6-4854-9aa3-b6bc501e43c8 2024-01-13T11:49:44.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:49:44.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:49:44.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:49:44.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:49:44.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:49:44.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:49:44.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:49:44.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:49:44.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:49:44.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:49:44.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:49:44.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:49:44.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:49:44.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:49:44.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:49:44.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:49:44.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:49:44.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:49:44.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:49:44.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:49:44.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:49:44.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:49:44.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:49:44.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:49:44.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:49:44.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:49:44.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:49:44.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:49:44.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:49:44.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:49:44.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:49:44.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:49:44.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:49:44.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:49:44.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:49:44.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:49:44.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:49:44.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:49:44.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:49:44.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:49:44.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:49:44.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:49:44.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:49:44.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:49:44.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:49:44.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:49:44.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:49:44.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:49:44.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:49:44.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:49:44.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:49:44.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:49:44.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:49:44.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:49:44.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:49:44.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:49:44.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:49:44.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:49:44.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: main() 2024-01-13T11:49:44.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:49:44.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:49:44.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:49:44.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:49:44.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:49:44.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:49:44.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:49:44.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:49:44.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:49:44.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:49:44.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:49:44.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:49:44.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:49:44.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:49:44.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmphoev5vri:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp0e8hz73y:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:49:44.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:49:44.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T11:49:44.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: if self._apply_service(spec): 2024-01-13T11:49:44.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T11:49:44.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T11:49:44.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T11:49:44.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T11:49:44.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T11:49:44.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T11:49:44.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T11:49:44.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T11:49:44.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T11:49:44.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: raise self._value 2024-01-13T11:49:44.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T11:49:44.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: result = (True, func(*args, **kwds)) 2024-01-13T11:49:44.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T11:49:44.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: return list(map(*args)) 2024-01-13T11:49:44.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:49:44.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: return f(*arg) 2024-01-13T11:49:44.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:49:44.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:49:44.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:49:44.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: code, '\n'.join(err))) 2024-01-13T11:49:44.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:49:44.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmphoev5vri:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp0e8hz73y:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:49:44.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:49:44.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:49:44.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:49:44.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new eb8b8805-c3d6-4854-9aa3-b6bc501e43c8 2024-01-13T11:49:44.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:49:44.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:49:44.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:49:44.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:49:44.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:49:44.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:49:44.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:49:44.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:49:44.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:49:44.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:49:44.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:49:44.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:49:44.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:49:44.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:49:44.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:49:44.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:49:44.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:49:44.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:49:44.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:49:44.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:49:44.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:49:44.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:49:44.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:49:44.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:49:44.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:49:44.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:49:44.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:49:44.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:49:44.398 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:49:44.398 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:49:44.398 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:49:44.398 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:49:44.398 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:49:44.398 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:49:44.398 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:49:44.398 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:49:44.398 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:49:44.398 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:49:44.398 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:49:44.398 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:49:44.398 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:49:44.398 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:49:44.399 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:49:44.399 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:49:44.399 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:49:44.399 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:49:44.399 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:49:44.399 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:49:44.399 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:49:44.399 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:49:44.399 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:49:44.399 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:49:44.399 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:49:44.399 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:49:44.399 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:49:44.399 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:49:44.400 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:49:44.400 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:49:44.400 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: main() 2024-01-13T11:49:44.400 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:49:44.400 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:49:44.400 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:49:44.400 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:49:44.400 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:49:44.400 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:49:44.400 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:49:44.400 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:49:44.400 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:49:44.400 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:49:44.401 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:49:44.401 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:49:44.401 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:49:44.401 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:49:44.401 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmphoev5vri:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp0e8hz73y:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:49:44.401 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: cluster 2024-01-13T11:49:42.548645+0000 mgr.smithi088.xwlxdh (mgr.14182) 90 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:44.401 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:43 smithi138 bash[17620]: cluster 2024-01-13T11:49:43.045486+0000 mon.smithi088 (mon.0) 296 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T11:49:44.796 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:49:45.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:44 smithi088 bash[14069]: audit 2024-01-13T11:49:44.806334+0000 mon.smithi088 (mon.0) 297 : audit [DBG] from='client.? 172.21.15.88:0/3904042100' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:49:45.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:44 smithi138 bash[17620]: audit 2024-01-13T11:49:44.806334+0000 mon.smithi088 (mon.0) 297 : audit [DBG] from='client.? 172.21.15.88:0/3904042100' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:49:45.565 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:49:46.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:45 smithi138 bash[17620]: cluster 2024-01-13T11:49:44.549677+0000 mgr.smithi088.xwlxdh (mgr.14182) 91 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:46.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:45 smithi088 bash[14069]: cluster 2024-01-13T11:49:44.549677+0000 mgr.smithi088.xwlxdh (mgr.14182) 91 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:46.567 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:49:48.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:47 smithi138 bash[17620]: cluster 2024-01-13T11:49:46.550629+0000 mgr.smithi088.xwlxdh (mgr.14182) 92 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:48.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:47 smithi088 bash[14069]: cluster 2024-01-13T11:49:46.550629+0000 mgr.smithi088.xwlxdh (mgr.14182) 92 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:49.219 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:48 smithi088 bash[14069]: audit 2024-01-13T11:49:47.958487+0000 mon.smithi088 (mon.0) 298 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:49:49.219 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:48 smithi088 bash[14069]: cluster 2024-01-13T11:49:48.551669+0000 mgr.smithi088.xwlxdh (mgr.14182) 93 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:49.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:48 smithi138 bash[17620]: audit 2024-01-13T11:49:47.958487+0000 mon.smithi088 (mon.0) 298 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:49:49.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:48 smithi138 bash[17620]: cluster 2024-01-13T11:49:48.551669+0000 mgr.smithi088.xwlxdh (mgr.14182) 93 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:49.571 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:49:50.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:49 smithi088 bash[14069]: audit 2024-01-13T11:49:49.581840+0000 mon.smithi088 (mon.0) 299 : audit [DBG] from='client.? 172.21.15.88:0/2816509960' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:49:50.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:49 smithi088 bash[14069]: audit 2024-01-13T11:49:49.664489+0000 mon.smithi088 (mon.0) 300 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:49:50.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:49 smithi088 bash[14069]: audit 2024-01-13T11:49:49.665677+0000 mon.smithi088 (mon.0) 301 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:49:50.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:49 smithi088 bash[14069]: audit 2024-01-13T11:49:49.666860+0000 mon.smithi088 (mon.0) 302 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T11:49:50.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:50 smithi088 bash[14069]: audit 2024-01-13T11:49:49.673372+0000 mon.smithi088 (mon.0) 303 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:49:50.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:50 smithi088 bash[14069]: cluster 2024-01-13T11:49:49.673850+0000 mgr.smithi088.xwlxdh (mgr.14182) 94 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:50.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:50 smithi088 bash[14069]: audit 2024-01-13T11:49:49.680723+0000 mon.smithi088 (mon.0) 304 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T11:49:50.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:50 smithi088 bash[14069]: audit 2024-01-13T11:49:49.685303+0000 mon.smithi088 (mon.0) 305 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:49:50.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:50 smithi088 bash[14069]: audit 2024-01-13T11:49:49.686996+0000 mon.smithi088 (mon.0) 306 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:49:50.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:50 smithi088 bash[14069]: audit 2024-01-13T11:49:49.687374+0000 mon.smithi088 (mon.0) 307 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:49:50.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:50 smithi088 bash[14069]: audit 2024-01-13T11:49:49.688203+0000 mon.smithi088 (mon.0) 308 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:49:50.368 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:49:50.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:49 smithi138 bash[17620]: audit 2024-01-13T11:49:49.581840+0000 mon.smithi088 (mon.0) 299 : audit [DBG] from='client.? 172.21.15.88:0/2816509960' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:49:50.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:49 smithi138 bash[17620]: audit 2024-01-13T11:49:49.664489+0000 mon.smithi088 (mon.0) 300 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:49:50.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:49 smithi138 bash[17620]: audit 2024-01-13T11:49:49.665677+0000 mon.smithi088 (mon.0) 301 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:49:50.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:49 smithi138 bash[17620]: audit 2024-01-13T11:49:49.666860+0000 mon.smithi088 (mon.0) 302 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T11:49:50.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:49 smithi138 bash[17620]: audit 2024-01-13T11:49:49.673372+0000 mon.smithi088 (mon.0) 303 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:49:50.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:49 smithi138 bash[17620]: cluster 2024-01-13T11:49:49.673850+0000 mgr.smithi088.xwlxdh (mgr.14182) 94 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:50.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:49 smithi138 bash[17620]: audit 2024-01-13T11:49:49.680723+0000 mon.smithi088 (mon.0) 304 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T11:49:50.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:49 smithi138 bash[17620]: audit 2024-01-13T11:49:49.685303+0000 mon.smithi088 (mon.0) 305 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:49:50.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:49 smithi138 bash[17620]: audit 2024-01-13T11:49:49.686996+0000 mon.smithi088 (mon.0) 306 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:49:50.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:49 smithi138 bash[17620]: audit 2024-01-13T11:49:49.687374+0000 mon.smithi088 (mon.0) 307 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:49:50.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:49 smithi138 bash[17620]: audit 2024-01-13T11:49:49.688203+0000 mon.smithi088 (mon.0) 308 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:49:51.369 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:49:51.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:50 smithi138 bash[17620]: cluster 2024-01-13T11:49:50.671413+0000 mon.smithi088 (mon.0) 309 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T11:49:51.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:50 smithi138 bash[17620]: cluster 2024-01-13T11:49:50.671471+0000 mon.smithi088 (mon.0) 310 : cluster [INF] Cluster is now healthy 2024-01-13T11:49:51.383 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:50 smithi088 bash[14069]: cluster 2024-01-13T11:49:50.671413+0000 mon.smithi088 (mon.0) 309 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T11:49:51.383 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:50 smithi088 bash[14069]: cluster 2024-01-13T11:49:50.671471+0000 mon.smithi088 (mon.0) 310 : cluster [INF] Cluster is now healthy 2024-01-13T11:49:52.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:51 smithi138 bash[17620]: cluster 2024-01-13T11:49:51.674297+0000 mgr.smithi088.xwlxdh (mgr.14182) 95 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:52.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:51 smithi088 bash[14069]: cluster 2024-01-13T11:49:51.674297+0000 mgr.smithi088.xwlxdh (mgr.14182) 95 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:54.603 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:49:55.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:54 smithi138 bash[17620]: cluster 2024-01-13T11:49:53.674753+0000 mgr.smithi088.xwlxdh (mgr.14182) 96 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:55.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:54 smithi138 bash[17620]: audit 2024-01-13T11:49:54.613411+0000 mon.smithi088 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.88:0/1747585831' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:49:55.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:54 smithi088 bash[14069]: cluster 2024-01-13T11:49:53.674753+0000 mgr.smithi088.xwlxdh (mgr.14182) 96 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:55.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:54 smithi088 bash[14069]: audit 2024-01-13T11:49:54.613411+0000 mon.smithi088 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.88:0/1747585831' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:49:55.747 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:49:56.748 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:49:57.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:56 smithi138 bash[17620]: cluster 2024-01-13T11:49:55.675054+0000 mgr.smithi088.xwlxdh (mgr.14182) 97 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:57.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:56 smithi088 bash[14069]: cluster 2024-01-13T11:49:55.675054+0000 mgr.smithi088.xwlxdh (mgr.14182) 97 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:59.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:58 smithi138 bash[17620]: cluster 2024-01-13T11:49:57.675333+0000 mgr.smithi088.xwlxdh (mgr.14182) 98 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:59.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:58 smithi138 bash[17620]: audit 2024-01-13T11:49:58.450326+0000 mon.smithi138 (mon.1) 6 : audit [INF] from='client.? 172.21.15.138:0/4023113063' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "486b07e8-54f0-4ffa-9bbf-5644a3c779e2"}]: dispatch 2024-01-13T11:49:59.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:58 smithi138 bash[17620]: audit 2024-01-13T11:49:58.451181+0000 mon.smithi088 (mon.0) 312 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "486b07e8-54f0-4ffa-9bbf-5644a3c779e2"}]: dispatch 2024-01-13T11:49:59.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:58 smithi138 bash[17620]: audit 2024-01-13T11:49:58.456774+0000 mon.smithi088 (mon.0) 313 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "486b07e8-54f0-4ffa-9bbf-5644a3c779e2"}]': finished 2024-01-13T11:49:59.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:58 smithi138 bash[17620]: cluster 2024-01-13T11:49:58.456918+0000 mon.smithi088 (mon.0) 314 : cluster [DBG] osdmap e14: 1 total, 0 up, 1 in 2024-01-13T11:49:59.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:58 smithi138 bash[17620]: audit 2024-01-13T11:49:58.457080+0000 mon.smithi088 (mon.0) 315 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:49:59.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:58 smithi138 bash[17620]: audit 2024-01-13T11:49:58.821685+0000 mon.smithi138 (mon.1) 7 : audit [INF] from='client.? 172.21.15.138:0/3386870785' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:49:59.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:58 smithi138 bash[17620]: audit 2024-01-13T11:49:58.822630+0000 mon.smithi088 (mon.0) 316 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:49:59.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:58 smithi138 bash[17620]: audit 2024-01-13T11:49:58.827697+0000 mon.smithi088 (mon.0) 317 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:49:59.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:49:58 smithi138 bash[17620]: cluster 2024-01-13T11:49:58.827786+0000 mon.smithi088 (mon.0) 318 : cluster [DBG] osdmap e15: 0 total, 0 up, 0 in 2024-01-13T11:49:59.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:58 smithi088 bash[14069]: cluster 2024-01-13T11:49:57.675333+0000 mgr.smithi088.xwlxdh (mgr.14182) 98 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:49:59.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:58 smithi088 bash[14069]: audit 2024-01-13T11:49:58.450326+0000 mon.smithi138 (mon.1) 6 : audit [INF] from='client.? 172.21.15.138:0/4023113063' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "486b07e8-54f0-4ffa-9bbf-5644a3c779e2"}]: dispatch 2024-01-13T11:49:59.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:58 smithi088 bash[14069]: audit 2024-01-13T11:49:58.451181+0000 mon.smithi088 (mon.0) 312 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "486b07e8-54f0-4ffa-9bbf-5644a3c779e2"}]: dispatch 2024-01-13T11:49:59.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:58 smithi088 bash[14069]: audit 2024-01-13T11:49:58.456774+0000 mon.smithi088 (mon.0) 313 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "486b07e8-54f0-4ffa-9bbf-5644a3c779e2"}]': finished 2024-01-13T11:49:59.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:58 smithi088 bash[14069]: cluster 2024-01-13T11:49:58.456918+0000 mon.smithi088 (mon.0) 314 : cluster [DBG] osdmap e14: 1 total, 0 up, 1 in 2024-01-13T11:49:59.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:58 smithi088 bash[14069]: audit 2024-01-13T11:49:58.457080+0000 mon.smithi088 (mon.0) 315 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:49:59.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:58 smithi088 bash[14069]: audit 2024-01-13T11:49:58.821685+0000 mon.smithi138 (mon.1) 7 : audit [INF] from='client.? 172.21.15.138:0/3386870785' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:49:59.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:58 smithi088 bash[14069]: audit 2024-01-13T11:49:58.822630+0000 mon.smithi088 (mon.0) 316 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:49:59.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:58 smithi088 bash[14069]: audit 2024-01-13T11:49:58.827697+0000 mon.smithi088 (mon.0) 317 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:49:59.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:49:58 smithi088 bash[14069]: cluster 2024-01-13T11:49:58.827786+0000 mon.smithi088 (mon.0) 318 : cluster [DBG] osdmap e15: 0 total, 0 up, 0 in 2024-01-13T11:50:00.390 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:50:00.892 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: cluster 2024-01-13T11:49:59.675718+0000 mgr.smithi088.xwlxdh (mgr.14182) 99 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:00.892 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: audit 2024-01-13T11:49:59.725012+0000 mon.smithi088 (mon.0) 319 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:50:00.893 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: cephadm 2024-01-13T11:49:59.757271+0000 mgr.smithi088.xwlxdh (mgr.14182) 100 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:50:00.893 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:50:00.893 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:50:00.893 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: return f(*arg) 2024-01-13T11:50:00.893 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:50:00.893 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:50:00.893 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:50:00.893 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: code, '\n'.join(err))) 2024-01-13T11:50:00.893 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:50:00.893 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp32naji3t:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp466n1vsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:50:00.893 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:50:00.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:50:00.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:50:00.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 486b07e8-54f0-4ffa-9bbf-5644a3c779e2 2024-01-13T11:50:00.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:50:00.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:50:00.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:50:00.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:50:00.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:50:00.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:50:00.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:50:00.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:50:00.894 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:50:00.895 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:50:00.895 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:50:00.895 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:50:00.895 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:50:00.895 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:50:00.895 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:50:00.895 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:50:00.895 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:50:00.895 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:50:00.895 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:50:00.895 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:50:00.895 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:50:00.896 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:50:00.896 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:50:00.896 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:50:00.896 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:50:00.896 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:50:00.896 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:50:00.896 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:50:00.896 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:50:00.896 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:50:00.896 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:50:00.896 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:50:00.896 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:50:00.897 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:50:00.897 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:50:00.897 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:50:00.897 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:50:00.897 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:50:00.897 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:50:00.897 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:50:00.897 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:50:00.897 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:50:00.897 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:50:00.897 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:50:00.897 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:50:00.898 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:50:00.898 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:50:00.898 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:50:00.898 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:50:00.898 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:50:00.898 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:50:00.898 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:50:00.898 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:50:00.898 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:50:00.898 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:50:00.898 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:50:00.898 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:50:00.899 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:50:00.899 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: main() 2024-01-13T11:50:00.899 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:50:00.899 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:50:00.899 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:50:00.899 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:50:00.899 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:50:00.899 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:50:00.899 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:50:00.899 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:50:00.899 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:50:00.899 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:50:00.900 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:50:00.900 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:50:00.900 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:50:00.900 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:50:00.900 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp32naji3t:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp466n1vsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:50:00.900 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: audit 2024-01-13T11:49:59.830239+0000 mon.smithi088 (mon.0) 320 : audit [INF] from='client.? 172.21.15.88:0/3596094262' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a03dcbc-7403-4675-8427-c28d95bd9033"}]: dispatch 2024-01-13T11:50:00.900 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: audit 2024-01-13T11:49:59.836920+0000 mon.smithi088 (mon.0) 321 : audit [INF] from='client.? 172.21.15.88:0/3596094262' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a03dcbc-7403-4675-8427-c28d95bd9033"}]': finished 2024-01-13T11:50:00.900 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: cluster 2024-01-13T11:49:59.837002+0000 mon.smithi088 (mon.0) 322 : cluster [DBG] osdmap e16: 1 total, 0 up, 1 in 2024-01-13T11:50:00.900 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: audit 2024-01-13T11:49:59.837141+0000 mon.smithi088 (mon.0) 323 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:50:00.900 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: cluster 2024-01-13T11:50:00.000124+0000 mon.smithi088 (mon.0) 324 : cluster [INF] overall HEALTH_OK 2024-01-13T11:50:00.900 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: audit 2024-01-13T11:50:00.190707+0000 mon.smithi088 (mon.0) 325 : audit [INF] from='client.? 172.21.15.88:0/2807332843' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:50:00.900 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: audit 2024-01-13T11:50:00.195743+0000 mon.smithi088 (mon.0) 326 : audit [INF] from='client.? 172.21.15.88:0/2807332843' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:50:00.901 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: cluster 2024-01-13T11:50:00.195797+0000 mon.smithi088 (mon.0) 327 : cluster [DBG] osdmap e17: 0 total, 0 up, 0 in 2024-01-13T11:50:00.901 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:00 smithi088 bash[14069]: audit 2024-01-13T11:50:00.400592+0000 mon.smithi088 (mon.0) 328 : audit [DBG] from='client.? 172.21.15.88:0/2736490928' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:50:01.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: cluster 2024-01-13T11:49:59.675718+0000 mgr.smithi088.xwlxdh (mgr.14182) 99 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:01.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: audit 2024-01-13T11:49:59.725012+0000 mon.smithi088 (mon.0) 319 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:50:01.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: cephadm 2024-01-13T11:49:59.757271+0000 mgr.smithi088.xwlxdh (mgr.14182) 100 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:50:01.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:50:01.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:50:01.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: return f(*arg) 2024-01-13T11:50:01.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:50:01.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:50:01.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:50:01.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: code, '\n'.join(err))) 2024-01-13T11:50:01.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:50:01.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp32naji3t:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp466n1vsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:50:01.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:50:01.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:50:01.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:50:01.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 486b07e8-54f0-4ffa-9bbf-5644a3c779e2 2024-01-13T11:50:01.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:50:01.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:50:01.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:50:01.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:50:01.127 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:50:01.127 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:50:01.127 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:50:01.127 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:50:01.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:50:01.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:50:01.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:50:01.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:50:01.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:50:01.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:50:01.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:50:01.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:50:01.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:50:01.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:50:01.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:50:01.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:50:01.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:50:01.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:50:01.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:50:01.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:50:01.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:50:01.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:50:01.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:50:01.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:50:01.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:50:01.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:50:01.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:50:01.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:50:01.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:50:01.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:50:01.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:50:01.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:50:01.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:50:01.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:50:01.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:50:01.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:50:01.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:50:01.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:50:01.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:50:01.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:50:01.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:50:01.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:50:01.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:50:01.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:50:01.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:50:01.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:50:01.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:50:01.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:50:01.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:50:01.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:50:01.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:50:01.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:50:01.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:50:01.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:50:01.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: main() 2024-01-13T11:50:01.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:50:01.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:50:01.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:50:01.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:50:01.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:50:01.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:50:01.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:50:01.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:50:01.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:50:01.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:50:01.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:50:01.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:50:01.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:50:01.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:50:01.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp32naji3t:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp466n1vsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:50:01.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: audit 2024-01-13T11:49:59.830239+0000 mon.smithi088 (mon.0) 320 : audit [INF] from='client.? 172.21.15.88:0/3596094262' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a03dcbc-7403-4675-8427-c28d95bd9033"}]: dispatch 2024-01-13T11:50:01.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: audit 2024-01-13T11:49:59.836920+0000 mon.smithi088 (mon.0) 321 : audit [INF] from='client.? 172.21.15.88:0/3596094262' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a03dcbc-7403-4675-8427-c28d95bd9033"}]': finished 2024-01-13T11:50:01.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: cluster 2024-01-13T11:49:59.837002+0000 mon.smithi088 (mon.0) 322 : cluster [DBG] osdmap e16: 1 total, 0 up, 1 in 2024-01-13T11:50:01.133 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: audit 2024-01-13T11:49:59.837141+0000 mon.smithi088 (mon.0) 323 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:50:01.133 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: cluster 2024-01-13T11:50:00.000124+0000 mon.smithi088 (mon.0) 324 : cluster [INF] overall HEALTH_OK 2024-01-13T11:50:01.133 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: audit 2024-01-13T11:50:00.190707+0000 mon.smithi088 (mon.0) 325 : audit [INF] from='client.? 172.21.15.88:0/2807332843' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:50:01.133 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: audit 2024-01-13T11:50:00.195743+0000 mon.smithi088 (mon.0) 326 : audit [INF] from='client.? 172.21.15.88:0/2807332843' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:50:01.133 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: cluster 2024-01-13T11:50:00.195797+0000 mon.smithi088 (mon.0) 327 : cluster [DBG] osdmap e17: 0 total, 0 up, 0 in 2024-01-13T11:50:01.133 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:00 smithi138 bash[17620]: audit 2024-01-13T11:50:00.400592+0000 mon.smithi088 (mon.0) 328 : audit [DBG] from='client.? 172.21.15.88:0/2736490928' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:50:01.629 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:50:01.892 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:01 smithi088 bash[14069]: cluster 2024-01-13T11:50:01.727381+0000 mon.smithi088 (mon.0) 329 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T11:50:02.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:01 smithi138 bash[17620]: cluster 2024-01-13T11:50:01.727381+0000 mon.smithi088 (mon.0) 329 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T11:50:02.630 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:50:02.875 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: cephadm 2024-01-13T11:50:01.606490+0000 mgr.smithi088.xwlxdh (mgr.14182) 101 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:50:02.875 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:50:02.875 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:50:02.875 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: return f(*arg) 2024-01-13T11:50:02.875 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:50:02.875 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:50:02.876 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:50:02.876 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: code, '\n'.join(err))) 2024-01-13T11:50:02.876 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi088/config 2024-01-13T11:50:02.876 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpvh02nugh:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp6er3k38p:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:50:02.876 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:50:02.876 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:50:02.876 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:50:02.876 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 7a03dcbc-7403-4675-8427-c28d95bd9033 2024-01-13T11:50:02.876 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:50:02.876 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:50:02.876 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:50:02.876 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:50:02.876 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:50:02.877 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:50:02.877 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:50:02.878 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:50:02.878 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:50:02.878 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:50:02.878 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:50:02.878 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:50:02.878 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:50:02.878 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:50:02.878 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:50:02.878 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:50:02.878 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:50:02.878 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:50:02.878 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:50:02.879 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:50:02.879 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:50:02.879 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:50:02.879 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:50:02.879 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:50:02.879 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:50:02.879 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:50:02.879 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:50:02.879 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:50:02.879 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:50:02.879 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:50:02.879 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:50:02.879 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:50:02.880 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:50:02.880 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:50:02.880 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:50:02.880 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:50:02.880 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:50:02.880 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:50:02.880 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:50:02.880 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:50:02.880 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:50:02.881 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:50:02.881 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:50:02.881 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:50:02.881 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:50:02.881 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:50:02.881 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:50:02.881 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:50:02.881 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:50:02.881 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:50:02.881 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:50:02.881 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:50:02.881 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:50:02.881 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:50:02.882 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:50:02.882 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:50:02.882 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:50:02.882 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:50:02.882 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: main() 2024-01-13T11:50:02.882 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:50:02.882 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:50:02.882 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:50:02.882 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:50:02.882 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:50:02.883 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:50:02.883 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:50:02.883 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:50:02.883 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:50:02.883 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:50:02.883 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:50:02.883 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:50:02.883 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:50:02.883 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:50:02.883 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpvh02nugh:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp6er3k38p:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:50:02.883 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: cephadm 2024-01-13T11:50:01.608641+0000 mgr.smithi088.xwlxdh (mgr.14182) 102 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T11:50:02.883 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: service_id: all-available-devices 2024-01-13T11:50:02.883 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: service_name: osd.all-available-devices 2024-01-13T11:50:02.884 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: placement: 2024-01-13T11:50:02.884 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: host_pattern: '*' 2024-01-13T11:50:02.884 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: spec: 2024-01-13T11:50:02.884 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: data_devices: 2024-01-13T11:50:02.884 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: all: true 2024-01-13T11:50:02.884 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: filter_logic: AND 2024-01-13T11:50:02.884 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: objectstore: bluestore 2024-01-13T11:50:02.884 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:50:02.884 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp32naji3t:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp466n1vsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:50:02.884 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:50:02.885 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:50:02.885 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:50:02.885 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 486b07e8-54f0-4ffa-9bbf-5644a3c779e2 2024-01-13T11:50:02.885 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:50:02.885 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:50:02.885 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:50:02.885 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:50:02.885 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:50:02.885 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:50:02.885 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:50:02.885 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:50:02.885 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:50:02.885 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:50:02.885 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:50:02.886 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:50:02.886 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:50:02.886 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:50:02.886 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:50:02.886 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:50:02.886 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:50:02.886 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:50:02.886 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:50:02.886 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:50:02.886 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:50:02.886 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:50:02.886 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:50:02.886 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:50:02.886 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:50:02.887 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:50:02.887 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:50:02.887 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:50:02.887 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:50:02.887 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:50:02.887 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:50:02.887 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:50:02.887 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:50:02.887 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:50:02.887 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:50:02.887 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:50:02.887 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:50:02.887 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:50:02.887 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:50:02.888 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:50:02.888 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:50:02.888 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:50:02.888 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:50:02.888 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:50:02.888 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:50:02.888 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:50:02.888 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:50:02.888 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:50:02.888 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:50:02.888 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:50:02.888 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:50:02.888 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:50:02.888 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:50:02.889 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:50:02.889 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:50:02.889 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:50:02.889 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:50:02.889 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:50:02.889 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: main() 2024-01-13T11:50:02.889 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:50:02.889 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:50:02.889 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:50:02.889 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:50:02.889 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:50:02.889 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:50:02.889 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:50:02.889 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:50:02.890 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:50:02.890 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:50:02.891 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:50:02.891 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:50:02.891 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:50:02.891 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:50:02.891 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp32naji3t:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp466n1vsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:50:02.891 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:50:02.891 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T11:50:02.891 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: if self._apply_service(spec): 2024-01-13T11:50:02.891 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T11:50:02.891 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T11:50:02.891 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T11:50:02.891 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T11:50:02.891 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T11:50:02.892 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T11:50:02.892 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T11:50:02.892 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T11:50:02.892 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T11:50:02.892 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: raise self._value 2024-01-13T11:50:02.892 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T11:50:02.892 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: result = (True, func(*args, **kwds)) 2024-01-13T11:50:02.892 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T11:50:02.892 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: return list(map(*args)) 2024-01-13T11:50:02.892 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:50:02.892 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: return f(*arg) 2024-01-13T11:50:02.892 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:50:02.892 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:50:02.893 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:50:02.893 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: code, '\n'.join(err))) 2024-01-13T11:50:02.893 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:50:02.893 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp32naji3t:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp466n1vsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:50:02.893 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:50:02.893 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:50:02.893 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:50:02.893 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 486b07e8-54f0-4ffa-9bbf-5644a3c779e2 2024-01-13T11:50:02.893 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:50:02.893 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:50:02.893 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:50:02.893 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:50:02.893 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:50:02.893 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:50:02.894 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:50:02.894 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:50:02.894 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:50:02.894 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:50:02.894 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:50:02.894 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:50:02.894 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:50:02.894 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:50:02.894 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:50:02.894 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:50:02.894 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:50:02.894 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:50:02.894 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:50:02.895 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:50:02.895 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:50:02.895 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:50:02.895 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:50:02.895 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:50:02.895 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:50:02.895 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:50:02.895 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:50:02.895 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:50:02.895 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:50:02.895 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:50:02.895 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:50:02.895 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:50:02.895 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:50:02.896 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:50:02.896 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:50:02.896 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:50:02.896 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:50:02.896 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:50:02.896 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:50:02.896 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:50:02.896 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:50:02.896 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:50:02.896 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:50:02.896 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:50:02.896 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:50:02.896 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:50:02.897 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:50:02.897 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:50:02.897 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:50:02.897 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:50:02.897 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:50:02.897 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:50:02.897 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:50:02.897 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:50:02.897 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:50:02.897 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:50:02.897 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:50:02.897 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:50:02.897 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: main() 2024-01-13T11:50:02.897 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:50:02.897 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:50:02.898 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:50:02.898 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:50:02.898 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:50:02.898 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:50:02.898 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:50:02.898 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:50:02.898 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:50:02.898 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:50:02.898 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:50:02.898 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:50:02.898 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:50:02.898 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:50:02.898 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp32naji3t:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp466n1vsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:50:02.898 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:02 smithi138 bash[17620]: cluster 2024-01-13T11:50:01.609461+0000 mgr.smithi088.xwlxdh (mgr.14182) 103 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:03.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: cephadm 2024-01-13T11:50:01.606490+0000 mgr.smithi088.xwlxdh (mgr.14182) 101 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:50:03.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:50:03.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:50:03.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: return f(*arg) 2024-01-13T11:50:03.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:50:03.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:50:03.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:50:03.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: code, '\n'.join(err))) 2024-01-13T11:50:03.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi088/config 2024-01-13T11:50:03.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpvh02nugh:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp6er3k38p:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:50:03.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:50:03.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:50:03.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:50:03.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 7a03dcbc-7403-4675-8427-c28d95bd9033 2024-01-13T11:50:03.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:50:03.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:50:03.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:50:03.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:50:03.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:50:03.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:50:03.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:50:03.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:50:03.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:50:03.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:50:03.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:50:03.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:50:03.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:50:03.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:50:03.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:50:03.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:50:03.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:50:03.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:50:03.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:50:03.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:50:03.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:50:03.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:50:03.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:50:03.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:50:03.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:50:03.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:50:03.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:50:03.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:50:03.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:50:03.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:50:03.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:50:03.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:50:03.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:50:03.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:50:03.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:50:03.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:50:03.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:50:03.147 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:50:03.147 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:50:03.147 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:50:03.147 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:50:03.147 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:50:03.147 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:50:03.147 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:50:03.147 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:50:03.147 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:50:03.147 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:50:03.147 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:50:03.147 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:50:03.147 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:50:03.147 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:50:03.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:50:03.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:50:03.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:50:03.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:50:03.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:50:03.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:50:03.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:50:03.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: main() 2024-01-13T11:50:03.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:50:03.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:50:03.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:50:03.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:50:03.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:50:03.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:50:03.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:50:03.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:50:03.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:50:03.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:50:03.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:50:03.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:50:03.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:50:03.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:50:03.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpvh02nugh:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp6er3k38p:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:50:03.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: cephadm 2024-01-13T11:50:01.608641+0000 mgr.smithi088.xwlxdh (mgr.14182) 102 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T11:50:03.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: service_id: all-available-devices 2024-01-13T11:50:03.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: service_name: osd.all-available-devices 2024-01-13T11:50:03.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: placement: 2024-01-13T11:50:03.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: host_pattern: '*' 2024-01-13T11:50:03.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: spec: 2024-01-13T11:50:03.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: data_devices: 2024-01-13T11:50:03.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: all: true 2024-01-13T11:50:03.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: filter_logic: AND 2024-01-13T11:50:03.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: objectstore: bluestore 2024-01-13T11:50:03.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:50:03.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp32naji3t:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp466n1vsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:50:03.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:50:03.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:50:03.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:50:03.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 486b07e8-54f0-4ffa-9bbf-5644a3c779e2 2024-01-13T11:50:03.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:50:03.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:50:03.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:50:03.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:50:03.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:50:03.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:50:03.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:50:03.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:50:03.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:50:03.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:50:03.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:50:03.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:50:03.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:50:03.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:50:03.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:50:03.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:50:03.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:50:03.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:50:03.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:50:03.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:50:03.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:50:03.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:50:03.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:50:03.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:50:03.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:50:03.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:50:03.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:50:03.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:50:03.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:50:03.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:50:03.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:50:03.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:50:03.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:50:03.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:50:03.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:50:03.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:50:03.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:50:03.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:50:03.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:50:03.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:50:03.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:50:03.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:50:03.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:50:03.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:50:03.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:50:03.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:50:03.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:50:03.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:50:03.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:50:03.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:50:03.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:50:03.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:50:03.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:50:03.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:50:03.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:50:03.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:50:03.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:50:03.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:50:03.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: main() 2024-01-13T11:50:03.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:50:03.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:50:03.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:50:03.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:50:03.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:50:03.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:50:03.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:50:03.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:50:03.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:50:03.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:50:03.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:50:03.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:50:03.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:50:03.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:50:03.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp32naji3t:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp466n1vsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:50:03.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:50:03.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T11:50:03.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: if self._apply_service(spec): 2024-01-13T11:50:03.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T11:50:03.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T11:50:03.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T11:50:03.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T11:50:03.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T11:50:03.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T11:50:03.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T11:50:03.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T11:50:03.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T11:50:03.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: raise self._value 2024-01-13T11:50:03.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T11:50:03.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: result = (True, func(*args, **kwds)) 2024-01-13T11:50:03.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T11:50:03.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: return list(map(*args)) 2024-01-13T11:50:03.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:50:03.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: return f(*arg) 2024-01-13T11:50:03.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:50:03.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:50:03.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:50:03.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: code, '\n'.join(err))) 2024-01-13T11:50:03.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:50:03.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp32naji3t:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp466n1vsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:50:03.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:50:03.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:50:03.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:50:03.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 486b07e8-54f0-4ffa-9bbf-5644a3c779e2 2024-01-13T11:50:03.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:50:03.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:50:03.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:50:03.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:50:03.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:50:03.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:50:03.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:50:03.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:50:03.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:50:03.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:50:03.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:50:03.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:50:03.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:50:03.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:50:03.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:50:03.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:50:03.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:50:03.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:50:03.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:50:03.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:50:03.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:50:03.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:50:03.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:50:03.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:50:03.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:50:03.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:50:03.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:50:03.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:50:03.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:50:03.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:50:03.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:50:03.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:50:03.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:50:03.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:50:03.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:50:03.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:50:03.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:50:03.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:50:03.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:50:03.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:50:03.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:50:03.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:50:03.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:50:03.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:50:03.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:50:03.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:50:03.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:50:03.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:50:03.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:50:03.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:50:03.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:50:03.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:50:03.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:50:03.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:50:03.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:50:03.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:50:03.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:50:03.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:50:03.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: main() 2024-01-13T11:50:03.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:50:03.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:50:03.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:50:03.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:50:03.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:50:03.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:50:03.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:50:03.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:50:03.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:50:03.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:50:03.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:50:03.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:50:03.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:50:03.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:50:03.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp32naji3t:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp466n1vsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:50:03.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:02 smithi088 bash[14069]: cluster 2024-01-13T11:50:01.609461+0000 mgr.smithi088.xwlxdh (mgr.14182) 103 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:05.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:04 smithi138 bash[17620]: cluster 2024-01-13T11:50:03.610472+0000 mgr.smithi088.xwlxdh (mgr.14182) 104 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:05.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:04 smithi088 bash[14069]: cluster 2024-01-13T11:50:03.610472+0000 mgr.smithi088.xwlxdh (mgr.14182) 104 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:05.736 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:50:06.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:05 smithi138 bash[17620]: audit 2024-01-13T11:50:05.746256+0000 mon.smithi088 (mon.0) 330 : audit [DBG] from='client.? 172.21.15.88:0/3388866561' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:50:06.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:05 smithi088 bash[14069]: audit 2024-01-13T11:50:05.746256+0000 mon.smithi088 (mon.0) 330 : audit [DBG] from='client.? 172.21.15.88:0/3388866561' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:50:06.464 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:50:07.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:06 smithi138 bash[17620]: cluster 2024-01-13T11:50:05.611476+0000 mgr.smithi088.xwlxdh (mgr.14182) 105 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:07.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:06 smithi088 bash[14069]: cluster 2024-01-13T11:50:05.611476+0000 mgr.smithi088.xwlxdh (mgr.14182) 105 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:07.465 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:50:09.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:08 smithi138 bash[17620]: cluster 2024-01-13T11:50:07.612089+0000 mgr.smithi088.xwlxdh (mgr.14182) 106 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:09.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:08 smithi088 bash[14069]: cluster 2024-01-13T11:50:07.612089+0000 mgr.smithi088.xwlxdh (mgr.14182) 106 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:10.607 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:50:11.123 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:10 smithi088 bash[14069]: cluster 2024-01-13T11:50:09.613104+0000 mgr.smithi088.xwlxdh (mgr.14182) 107 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:11.123 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:10 smithi088 bash[14069]: audit 2024-01-13T11:50:10.617461+0000 mon.smithi088 (mon.0) 331 : audit [DBG] from='client.? 172.21.15.88:0/3348374705' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:50:11.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:10 smithi138 bash[17620]: cluster 2024-01-13T11:50:09.613104+0000 mgr.smithi088.xwlxdh (mgr.14182) 107 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:11.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:10 smithi138 bash[17620]: audit 2024-01-13T11:50:10.617461+0000 mon.smithi088 (mon.0) 331 : audit [DBG] from='client.? 172.21.15.88:0/3348374705' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:50:11.441 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:50:12.442 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:50:12.891 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:12 smithi088 bash[14069]: cluster 2024-01-13T11:50:11.614138+0000 mgr.smithi088.xwlxdh (mgr.14182) 108 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:13.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:12 smithi138 bash[17620]: cluster 2024-01-13T11:50:11.614138+0000 mgr.smithi088.xwlxdh (mgr.14182) 108 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:15.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:14 smithi088 bash[14069]: cluster 2024-01-13T11:50:13.615233+0000 mgr.smithi088.xwlxdh (mgr.14182) 109 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:15.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:14 smithi088 bash[14069]: audit 2024-01-13T11:50:14.726460+0000 mon.smithi088 (mon.0) 332 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:50:15.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:14 smithi138 bash[17620]: cluster 2024-01-13T11:50:13.615233+0000 mgr.smithi088.xwlxdh (mgr.14182) 109 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:15.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:14 smithi138 bash[17620]: audit 2024-01-13T11:50:14.726460+0000 mon.smithi088 (mon.0) 332 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:50:15.518 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:50:16.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:15 smithi088 bash[14069]: audit 2024-01-13T11:50:15.529202+0000 mon.smithi088 (mon.0) 333 : audit [DBG] from='client.? 172.21.15.88:0/1089810325' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:50:16.327 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:50:16.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:15 smithi138 bash[17620]: audit 2024-01-13T11:50:15.529202+0000 mon.smithi088 (mon.0) 333 : audit [DBG] from='client.? 172.21.15.88:0/1089810325' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:50:17.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:16 smithi088 bash[14069]: cluster 2024-01-13T11:50:15.616310+0000 mgr.smithi088.xwlxdh (mgr.14182) 110 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:17.328 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:50:17.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:16 smithi138 bash[17620]: cluster 2024-01-13T11:50:15.616310+0000 mgr.smithi088.xwlxdh (mgr.14182) 110 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:19.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:18 smithi088 bash[14069]: cluster 2024-01-13T11:50:17.617359+0000 mgr.smithi088.xwlxdh (mgr.14182) 111 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:19.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:18 smithi138 bash[17620]: cluster 2024-01-13T11:50:17.617359+0000 mgr.smithi088.xwlxdh (mgr.14182) 111 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:20.445 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:50:21.121 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:20 smithi088 bash[14069]: cluster 2024-01-13T11:50:19.617906+0000 mgr.smithi088.xwlxdh (mgr.14182) 112 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:21.121 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:20 smithi088 bash[14069]: audit 2024-01-13T11:50:20.455797+0000 mon.smithi088 (mon.0) 334 : audit [DBG] from='client.? 172.21.15.88:0/573614757' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:50:21.161 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:50:21.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:20 smithi138 bash[17620]: cluster 2024-01-13T11:50:19.617906+0000 mgr.smithi088.xwlxdh (mgr.14182) 112 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:21.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:20 smithi138 bash[17620]: audit 2024-01-13T11:50:20.455797+0000 mon.smithi088 (mon.0) 334 : audit [DBG] from='client.? 172.21.15.88:0/573614757' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:50:22.163 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:50:23.223 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:22 smithi088 bash[14069]: cluster 2024-01-13T11:50:21.618961+0000 mgr.smithi088.xwlxdh (mgr.14182) 113 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:23.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:22 smithi138 bash[17620]: cluster 2024-01-13T11:50:21.618961+0000 mgr.smithi088.xwlxdh (mgr.14182) 113 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:25.318 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:50:25.369 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:24 smithi088 bash[14069]: cluster 2024-01-13T11:50:23.619466+0000 mgr.smithi088.xwlxdh (mgr.14182) 114 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:25.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:24 smithi138 bash[17620]: cluster 2024-01-13T11:50:23.619466+0000 mgr.smithi088.xwlxdh (mgr.14182) 114 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:26.082 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:50:26.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:25 smithi138 bash[17620]: audit 2024-01-13T11:50:25.328274+0000 mon.smithi088 (mon.0) 335 : audit [DBG] from='client.? 172.21.15.88:0/882905251' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:50:26.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:25 smithi088 bash[14069]: audit 2024-01-13T11:50:25.328274+0000 mon.smithi088 (mon.0) 335 : audit [DBG] from='client.? 172.21.15.88:0/882905251' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:50:27.084 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:50:27.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:26 smithi138 bash[17620]: cluster 2024-01-13T11:50:25.620370+0000 mgr.smithi088.xwlxdh (mgr.14182) 115 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:27.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:26 smithi088 bash[14069]: cluster 2024-01-13T11:50:25.620370+0000 mgr.smithi088.xwlxdh (mgr.14182) 115 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:29.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:28 smithi138 bash[17620]: cluster 2024-01-13T11:50:27.621181+0000 mgr.smithi088.xwlxdh (mgr.14182) 116 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:29.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:28 smithi088 bash[14069]: cluster 2024-01-13T11:50:27.621181+0000 mgr.smithi088.xwlxdh (mgr.14182) 116 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:29.987 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:50:30.263 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:30 smithi088 bash[14069]: cluster 2024-01-13T11:50:29.622376+0000 mgr.smithi088.xwlxdh (mgr.14182) 117 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:30.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:30 smithi138 bash[17620]: cluster 2024-01-13T11:50:29.622376+0000 mgr.smithi088.xwlxdh (mgr.14182) 117 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:30.734 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:50:31.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:31 smithi138 bash[17620]: audit 2024-01-13T11:50:29.997871+0000 mon.smithi088 (mon.0) 336 : audit [DBG] from='client.? 172.21.15.88:0/1454172242' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:50:31.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:31 smithi088 bash[14069]: audit 2024-01-13T11:50:29.997871+0000 mon.smithi088 (mon.0) 336 : audit [DBG] from='client.? 172.21.15.88:0/1454172242' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:50:31.736 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:50:32.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:32 smithi138 bash[17620]: cluster 2024-01-13T11:50:31.623423+0000 mgr.smithi088.xwlxdh (mgr.14182) 118 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:32.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:32 smithi088 bash[14069]: cluster 2024-01-13T11:50:31.623423+0000 mgr.smithi088.xwlxdh (mgr.14182) 118 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:34.945 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:50:35.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:34 smithi138 bash[17620]: cluster 2024-01-13T11:50:33.624515+0000 mgr.smithi088.xwlxdh (mgr.14182) 119 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:35.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:34 smithi088 bash[14069]: cluster 2024-01-13T11:50:33.624515+0000 mgr.smithi088.xwlxdh (mgr.14182) 119 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:35.678 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:50:36.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:35 smithi138 bash[17620]: audit 2024-01-13T11:50:34.955756+0000 mon.smithi088 (mon.0) 337 : audit [DBG] from='client.? 172.21.15.88:0/1380940716' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:50:36.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:35 smithi088 bash[14069]: audit 2024-01-13T11:50:34.955756+0000 mon.smithi088 (mon.0) 337 : audit [DBG] from='client.? 172.21.15.88:0/1380940716' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:50:36.679 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:50:37.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:36 smithi138 bash[17620]: cluster 2024-01-13T11:50:35.625600+0000 mgr.smithi088.xwlxdh (mgr.14182) 120 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:37.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:36 smithi088 bash[14069]: cluster 2024-01-13T11:50:35.625600+0000 mgr.smithi088.xwlxdh (mgr.14182) 120 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:39.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:38 smithi138 bash[17620]: cluster 2024-01-13T11:50:37.626798+0000 mgr.smithi088.xwlxdh (mgr.14182) 121 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:39.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:38 smithi088 bash[14069]: cluster 2024-01-13T11:50:37.626798+0000 mgr.smithi088.xwlxdh (mgr.14182) 121 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:39.481 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:50:40.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:39 smithi088 bash[14069]: audit 2024-01-13T11:50:39.492087+0000 mon.smithi088 (mon.0) 338 : audit [DBG] from='client.? 172.21.15.88:0/359459734' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:50:40.231 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:50:40.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:39 smithi138 bash[17620]: audit 2024-01-13T11:50:39.492087+0000 mon.smithi088 (mon.0) 338 : audit [DBG] from='client.? 172.21.15.88:0/359459734' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:50:41.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:40 smithi088 bash[14069]: cluster 2024-01-13T11:50:39.627854+0000 mgr.smithi088.xwlxdh (mgr.14182) 122 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:41.232 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:50:41.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:40 smithi138 bash[17620]: cluster 2024-01-13T11:50:39.627854+0000 mgr.smithi088.xwlxdh (mgr.14182) 122 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:43.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:42 smithi088 bash[14069]: cluster 2024-01-13T11:50:41.628904+0000 mgr.smithi088.xwlxdh (mgr.14182) 123 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:43.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:42 smithi138 bash[17620]: cluster 2024-01-13T11:50:41.628904+0000 mgr.smithi088.xwlxdh (mgr.14182) 123 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:44.421 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:50:45.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:44 smithi088 bash[14069]: cluster 2024-01-13T11:50:43.629699+0000 mgr.smithi088.xwlxdh (mgr.14182) 124 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:45.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:44 smithi088 bash[14069]: audit 2024-01-13T11:50:44.431507+0000 mon.smithi088 (mon.0) 339 : audit [DBG] from='client.? 172.21.15.88:0/2459173695' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:50:45.161 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:50:45.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:44 smithi138 bash[17620]: cluster 2024-01-13T11:50:43.629699+0000 mgr.smithi088.xwlxdh (mgr.14182) 124 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:45.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:44 smithi138 bash[17620]: audit 2024-01-13T11:50:44.431507+0000 mon.smithi088 (mon.0) 339 : audit [DBG] from='client.? 172.21.15.88:0/2459173695' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:50:46.162 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:50:47.243 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:46 smithi088 bash[14069]: cluster 2024-01-13T11:50:45.630775+0000 mgr.smithi088.xwlxdh (mgr.14182) 125 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:47.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:46 smithi138 bash[17620]: cluster 2024-01-13T11:50:45.630775+0000 mgr.smithi088.xwlxdh (mgr.14182) 125 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:49.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:48 smithi088 bash[14069]: cluster 2024-01-13T11:50:47.631860+0000 mgr.smithi088.xwlxdh (mgr.14182) 126 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:49.244 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:50:49.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:48 smithi138 bash[17620]: cluster 2024-01-13T11:50:47.631860+0000 mgr.smithi088.xwlxdh (mgr.14182) 126 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:49.982 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:50:50.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:49 smithi138 bash[17620]: audit 2024-01-13T11:50:49.255247+0000 mon.smithi088 (mon.0) 340 : audit [DBG] from='client.? 172.21.15.88:0/2279453792' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:50:50.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:49 smithi088 bash[14069]: audit 2024-01-13T11:50:49.255247+0000 mon.smithi088 (mon.0) 340 : audit [DBG] from='client.? 172.21.15.88:0/2279453792' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:50:50.983 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:50:51.178 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:50 smithi088 bash[14069]: cluster 2024-01-13T11:50:49.632657+0000 mgr.smithi088.xwlxdh (mgr.14182) 127 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:51.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:50 smithi138 bash[17620]: cluster 2024-01-13T11:50:49.632657+0000 mgr.smithi088.xwlxdh (mgr.14182) 127 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:53.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:52 smithi138 bash[17620]: cluster 2024-01-13T11:50:51.633410+0000 mgr.smithi088.xwlxdh (mgr.14182) 128 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:53.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:52 smithi088 bash[14069]: cluster 2024-01-13T11:50:51.633410+0000 mgr.smithi088.xwlxdh (mgr.14182) 128 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:54.210 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:50:54.969 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:50:55.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:54 smithi138 bash[17620]: cluster 2024-01-13T11:50:53.634037+0000 mgr.smithi088.xwlxdh (mgr.14182) 129 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:55.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:54 smithi138 bash[17620]: audit 2024-01-13T11:50:54.220537+0000 mon.smithi088 (mon.0) 341 : audit [DBG] from='client.? 172.21.15.88:0/1493658081' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:50:55.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:54 smithi088 bash[14069]: cluster 2024-01-13T11:50:53.634037+0000 mgr.smithi088.xwlxdh (mgr.14182) 129 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:55.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:54 smithi088 bash[14069]: audit 2024-01-13T11:50:54.220537+0000 mon.smithi088 (mon.0) 341 : audit [DBG] from='client.? 172.21.15.88:0/1493658081' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:50:55.971 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:50:57.193 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:56 smithi088 bash[14069]: cluster 2024-01-13T11:50:55.635036+0000 mgr.smithi088.xwlxdh (mgr.14182) 130 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:57.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:56 smithi138 bash[17620]: cluster 2024-01-13T11:50:55.635036+0000 mgr.smithi088.xwlxdh (mgr.14182) 130 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:58.891 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:58 smithi088 bash[14069]: cluster 2024-01-13T11:50:57.635676+0000 mgr.smithi088.xwlxdh (mgr.14182) 131 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:59.008 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:50:59.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:58 smithi138 bash[17620]: cluster 2024-01-13T11:50:57.635676+0000 mgr.smithi088.xwlxdh (mgr.14182) 131 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:50:59.672 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:51:00.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:50:59 smithi138 bash[17620]: audit 2024-01-13T11:50:59.018713+0000 mon.smithi088 (mon.0) 342 : audit [DBG] from='client.? 172.21.15.88:0/420207199' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:51:00.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:50:59 smithi088 bash[14069]: audit 2024-01-13T11:50:59.018713+0000 mon.smithi088 (mon.0) 342 : audit [DBG] from='client.? 172.21.15.88:0/420207199' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:51:00.674 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:51:01.122 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:00 smithi088 bash[14069]: cluster 2024-01-13T11:50:59.636649+0000 mgr.smithi088.xwlxdh (mgr.14182) 132 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:01.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:00 smithi138 bash[17620]: cluster 2024-01-13T11:50:59.636649+0000 mgr.smithi088.xwlxdh (mgr.14182) 132 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:02.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:01 smithi088 bash[14069]: audit 2024-01-13T11:51:01.611930+0000 mon.smithi088 (mon.0) 343 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:51:02.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:01 smithi138 bash[17620]: audit 2024-01-13T11:51:01.611930+0000 mon.smithi088 (mon.0) 343 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:51:03.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:02 smithi088 bash[14069]: cluster 2024-01-13T11:51:01.637647+0000 mgr.smithi088.xwlxdh (mgr.14182) 133 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:03.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:02 smithi138 bash[17620]: cluster 2024-01-13T11:51:01.637647+0000 mgr.smithi088.xwlxdh (mgr.14182) 133 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:03.878 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:51:04.138 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:03 smithi088 bash[14069]: audit 2024-01-13T11:51:03.889101+0000 mon.smithi088 (mon.0) 344 : audit [DBG] from='client.? 172.21.15.88:0/525866438' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:51:04.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:03 smithi138 bash[17620]: audit 2024-01-13T11:51:03.889101+0000 mon.smithi088 (mon.0) 344 : audit [DBG] from='client.? 172.21.15.88:0/525866438' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:51:04.624 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:51:05.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:04 smithi138 bash[17620]: cluster 2024-01-13T11:51:03.638535+0000 mgr.smithi088.xwlxdh (mgr.14182) 134 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:05.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:04 smithi088 bash[14069]: cluster 2024-01-13T11:51:03.638535+0000 mgr.smithi088.xwlxdh (mgr.14182) 134 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:05.625 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:51:07.045 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:06 smithi088 bash[14069]: cluster 2024-01-13T11:51:05.639711+0000 mgr.smithi088.xwlxdh (mgr.14182) 135 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:07.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:06 smithi138 bash[17620]: cluster 2024-01-13T11:51:05.639711+0000 mgr.smithi088.xwlxdh (mgr.14182) 135 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:08.299 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:08 smithi088 bash[14069]: audit 2024-01-13T11:51:07.053721+0000 mon.smithi088 (mon.0) 345 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:51:08.299 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:08 smithi088 bash[14069]: audit 2024-01-13T11:51:07.319562+0000 mon.smithi088 (mon.0) 346 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:51:08.299 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:08 smithi088 bash[14069]: audit 2024-01-13T11:51:07.321155+0000 mon.smithi088 (mon.0) 347 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T11:51:08.300 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:08 smithi088 bash[14069]: audit 2024-01-13T11:51:07.329650+0000 mon.smithi088 (mon.0) 348 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:51:08.300 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:08 smithi088 bash[14069]: cluster 2024-01-13T11:51:07.330302+0000 mgr.smithi088.xwlxdh (mgr.14182) 136 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:08.300 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:08 smithi088 bash[14069]: audit 2024-01-13T11:51:07.337124+0000 mon.smithi088 (mon.0) 349 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T11:51:08.300 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:08 smithi088 bash[14069]: audit 2024-01-13T11:51:07.341892+0000 mon.smithi088 (mon.0) 350 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:51:08.300 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:08 smithi088 bash[14069]: audit 2024-01-13T11:51:07.343263+0000 mon.smithi088 (mon.0) 351 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:51:08.300 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:08 smithi088 bash[14069]: audit 2024-01-13T11:51:07.343667+0000 mon.smithi088 (mon.0) 352 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:51:08.300 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:08 smithi088 bash[14069]: audit 2024-01-13T11:51:07.344599+0000 mon.smithi088 (mon.0) 353 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:51:08.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:08 smithi138 bash[17620]: audit 2024-01-13T11:51:07.053721+0000 mon.smithi088 (mon.0) 345 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:51:08.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:08 smithi138 bash[17620]: audit 2024-01-13T11:51:07.319562+0000 mon.smithi088 (mon.0) 346 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:51:08.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:08 smithi138 bash[17620]: audit 2024-01-13T11:51:07.321155+0000 mon.smithi088 (mon.0) 347 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T11:51:08.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:08 smithi138 bash[17620]: audit 2024-01-13T11:51:07.329650+0000 mon.smithi088 (mon.0) 348 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:51:08.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:08 smithi138 bash[17620]: cluster 2024-01-13T11:51:07.330302+0000 mgr.smithi088.xwlxdh (mgr.14182) 136 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:08.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:08 smithi138 bash[17620]: audit 2024-01-13T11:51:07.337124+0000 mon.smithi088 (mon.0) 349 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T11:51:08.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:08 smithi138 bash[17620]: audit 2024-01-13T11:51:07.341892+0000 mon.smithi088 (mon.0) 350 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:51:08.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:08 smithi138 bash[17620]: audit 2024-01-13T11:51:07.343263+0000 mon.smithi088 (mon.0) 351 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:51:08.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:08 smithi138 bash[17620]: audit 2024-01-13T11:51:07.343667+0000 mon.smithi088 (mon.0) 352 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:51:08.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:08 smithi138 bash[17620]: audit 2024-01-13T11:51:07.344599+0000 mon.smithi088 (mon.0) 353 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:51:08.710 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:51:09.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:09 smithi138 bash[17620]: cluster 2024-01-13T11:51:08.326455+0000 mon.smithi088 (mon.0) 354 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T11:51:09.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:09 smithi138 bash[17620]: cluster 2024-01-13T11:51:08.326522+0000 mon.smithi088 (mon.0) 355 : cluster [INF] Cluster is now healthy 2024-01-13T11:51:09.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:09 smithi138 bash[17620]: audit 2024-01-13T11:51:08.721127+0000 mon.smithi088 (mon.0) 356 : audit [DBG] from='client.? 172.21.15.88:0/1848440645' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:51:09.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:09 smithi088 bash[14069]: cluster 2024-01-13T11:51:08.326455+0000 mon.smithi088 (mon.0) 354 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T11:51:09.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:09 smithi088 bash[14069]: cluster 2024-01-13T11:51:08.326522+0000 mon.smithi088 (mon.0) 355 : cluster [INF] Cluster is now healthy 2024-01-13T11:51:09.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:09 smithi088 bash[14069]: audit 2024-01-13T11:51:08.721127+0000 mon.smithi088 (mon.0) 356 : audit [DBG] from='client.? 172.21.15.88:0/1848440645' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:51:09.489 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:51:10.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:10 smithi138 bash[17620]: cluster 2024-01-13T11:51:09.330668+0000 mgr.smithi088.xwlxdh (mgr.14182) 137 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:10.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:10 smithi088 bash[14069]: cluster 2024-01-13T11:51:09.330668+0000 mgr.smithi088.xwlxdh (mgr.14182) 137 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:10.490 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:51:12.641 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:12 smithi088 bash[14069]: cluster 2024-01-13T11:51:11.331128+0000 mgr.smithi088.xwlxdh (mgr.14182) 138 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:12.807 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:12 smithi138 bash[17620]: cluster 2024-01-13T11:51:11.331128+0000 mgr.smithi088.xwlxdh (mgr.14182) 138 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:14.453 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:51:15.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:14 smithi138 bash[17620]: cluster 2024-01-13T11:51:13.331506+0000 mgr.smithi088.xwlxdh (mgr.14182) 139 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:15.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:14 smithi138 bash[17620]: audit 2024-01-13T11:51:14.464221+0000 mon.smithi088 (mon.0) 357 : audit [DBG] from='client.? 172.21.15.88:0/893214411' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:51:15.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:14 smithi138 bash[17620]: audit 2024-01-13T11:51:14.729809+0000 mon.smithi088 (mon.0) 358 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:51:15.166 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:14 smithi088 bash[14069]: cluster 2024-01-13T11:51:13.331506+0000 mgr.smithi088.xwlxdh (mgr.14182) 139 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:15.166 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:14 smithi088 bash[14069]: audit 2024-01-13T11:51:14.464221+0000 mon.smithi088 (mon.0) 357 : audit [DBG] from='client.? 172.21.15.88:0/893214411' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:51:15.167 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:14 smithi088 bash[14069]: audit 2024-01-13T11:51:14.729809+0000 mon.smithi088 (mon.0) 358 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:51:15.860 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:51:16.861 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:51:17.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:16 smithi138 bash[17620]: cluster 2024-01-13T11:51:15.331842+0000 mgr.smithi088.xwlxdh (mgr.14182) 140 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:17.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:16 smithi138 bash[17620]: audit 2024-01-13T11:51:15.966710+0000 mon.smithi138 (mon.1) 8 : audit [INF] from='client.? 172.21.15.138:0/3235642148' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c0937fb-b80d-40a7-87f1-1a65720cab2a"}]: dispatch 2024-01-13T11:51:17.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:16 smithi138 bash[17620]: audit 2024-01-13T11:51:15.967504+0000 mon.smithi088 (mon.0) 359 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c0937fb-b80d-40a7-87f1-1a65720cab2a"}]: dispatch 2024-01-13T11:51:17.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:16 smithi138 bash[17620]: audit 2024-01-13T11:51:15.973225+0000 mon.smithi088 (mon.0) 360 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c0937fb-b80d-40a7-87f1-1a65720cab2a"}]': finished 2024-01-13T11:51:17.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:16 smithi138 bash[17620]: cluster 2024-01-13T11:51:15.973290+0000 mon.smithi088 (mon.0) 361 : cluster [DBG] osdmap e18: 1 total, 0 up, 1 in 2024-01-13T11:51:17.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:16 smithi138 bash[17620]: audit 2024-01-13T11:51:15.973533+0000 mon.smithi088 (mon.0) 362 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:51:17.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:16 smithi138 bash[17620]: audit 2024-01-13T11:51:16.317832+0000 mon.smithi138 (mon.1) 9 : audit [INF] from='client.? 172.21.15.138:0/4059429037' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:51:17.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:16 smithi138 bash[17620]: audit 2024-01-13T11:51:16.318768+0000 mon.smithi088 (mon.0) 363 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:51:17.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:16 smithi138 bash[17620]: audit 2024-01-13T11:51:16.325146+0000 mon.smithi088 (mon.0) 364 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:51:17.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:16 smithi138 bash[17620]: cluster 2024-01-13T11:51:16.325245+0000 mon.smithi088 (mon.0) 365 : cluster [DBG] osdmap e19: 0 total, 0 up, 0 in 2024-01-13T11:51:17.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:16 smithi088 bash[14069]: cluster 2024-01-13T11:51:15.331842+0000 mgr.smithi088.xwlxdh (mgr.14182) 140 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:17.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:16 smithi088 bash[14069]: audit 2024-01-13T11:51:15.966710+0000 mon.smithi138 (mon.1) 8 : audit [INF] from='client.? 172.21.15.138:0/3235642148' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c0937fb-b80d-40a7-87f1-1a65720cab2a"}]: dispatch 2024-01-13T11:51:17.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:16 smithi088 bash[14069]: audit 2024-01-13T11:51:15.967504+0000 mon.smithi088 (mon.0) 359 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c0937fb-b80d-40a7-87f1-1a65720cab2a"}]: dispatch 2024-01-13T11:51:17.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:16 smithi088 bash[14069]: audit 2024-01-13T11:51:15.973225+0000 mon.smithi088 (mon.0) 360 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c0937fb-b80d-40a7-87f1-1a65720cab2a"}]': finished 2024-01-13T11:51:17.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:16 smithi088 bash[14069]: cluster 2024-01-13T11:51:15.973290+0000 mon.smithi088 (mon.0) 361 : cluster [DBG] osdmap e18: 1 total, 0 up, 1 in 2024-01-13T11:51:17.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:16 smithi088 bash[14069]: audit 2024-01-13T11:51:15.973533+0000 mon.smithi088 (mon.0) 362 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:51:17.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:16 smithi088 bash[14069]: audit 2024-01-13T11:51:16.317832+0000 mon.smithi138 (mon.1) 9 : audit [INF] from='client.? 172.21.15.138:0/4059429037' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:51:17.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:16 smithi088 bash[14069]: audit 2024-01-13T11:51:16.318768+0000 mon.smithi088 (mon.0) 363 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:51:17.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:16 smithi088 bash[14069]: audit 2024-01-13T11:51:16.325146+0000 mon.smithi088 (mon.0) 364 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:51:17.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:16 smithi088 bash[14069]: cluster 2024-01-13T11:51:16.325245+0000 mon.smithi088 (mon.0) 365 : cluster [DBG] osdmap e19: 0 total, 0 up, 0 in 2024-01-13T11:51:18.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:17 smithi088 bash[14069]: audit 2024-01-13T11:51:17.762280+0000 mon.smithi088 (mon.0) 366 : audit [INF] from='client.? 172.21.15.88:0/2810624540' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b5463224-51fb-4da9-8869-52eae92055e5"}]: dispatch 2024-01-13T11:51:18.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:17 smithi088 bash[14069]: audit 2024-01-13T11:51:17.768460+0000 mon.smithi088 (mon.0) 367 : audit [INF] from='client.? 172.21.15.88:0/2810624540' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b5463224-51fb-4da9-8869-52eae92055e5"}]': finished 2024-01-13T11:51:18.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:17 smithi088 bash[14069]: cluster 2024-01-13T11:51:17.768510+0000 mon.smithi088 (mon.0) 368 : cluster [DBG] osdmap e20: 1 total, 0 up, 1 in 2024-01-13T11:51:18.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:17 smithi088 bash[14069]: audit 2024-01-13T11:51:17.768714+0000 mon.smithi088 (mon.0) 369 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:51:18.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:17 smithi138 bash[17620]: audit 2024-01-13T11:51:17.762280+0000 mon.smithi088 (mon.0) 366 : audit [INF] from='client.? 172.21.15.88:0/2810624540' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b5463224-51fb-4da9-8869-52eae92055e5"}]: dispatch 2024-01-13T11:51:18.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:17 smithi138 bash[17620]: audit 2024-01-13T11:51:17.768460+0000 mon.smithi088 (mon.0) 367 : audit [INF] from='client.? 172.21.15.88:0/2810624540' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b5463224-51fb-4da9-8869-52eae92055e5"}]': finished 2024-01-13T11:51:18.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:17 smithi138 bash[17620]: cluster 2024-01-13T11:51:17.768510+0000 mon.smithi088 (mon.0) 368 : cluster [DBG] osdmap e20: 1 total, 0 up, 1 in 2024-01-13T11:51:18.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:17 smithi138 bash[17620]: audit 2024-01-13T11:51:17.768714+0000 mon.smithi088 (mon.0) 369 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:51:18.928 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: cephadm 2024-01-13T11:51:17.237706+0000 mgr.smithi088.xwlxdh (mgr.14182) 141 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:51:18.929 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:51:18.929 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:51:18.929 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: return f(*arg) 2024-01-13T11:51:18.929 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:51:18.929 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:51:18.929 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:51:18.929 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: code, '\n'.join(err))) 2024-01-13T11:51:18.929 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:51:18.929 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp4a1qi124:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmph81obn55:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:51:18.929 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:51:18.929 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:51:18.930 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:51:18.930 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4c0937fb-b80d-40a7-87f1-1a65720cab2a 2024-01-13T11:51:19.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: cephadm 2024-01-13T11:51:17.237706+0000 mgr.smithi088.xwlxdh (mgr.14182) 141 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:51:19.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:51:19.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:51:19.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: return f(*arg) 2024-01-13T11:51:19.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:51:19.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:51:19.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:51:19.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: code, '\n'.join(err))) 2024-01-13T11:51:19.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:51:19.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp4a1qi124:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmph81obn55:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:51:19.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:51:19.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:51:19.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:51:19.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4c0937fb-b80d-40a7-87f1-1a65720cab2a 2024-01-13T11:51:19.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:51:19.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:51:19.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:51:19.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:51:19.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:51:19.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:51:19.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:51:19.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:51:19.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:51:19.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:51:19.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:51:19.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:51:19.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:51:19.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:51:19.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:51:19.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:51:19.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:51:19.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:51:19.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:51:19.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:51:19.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:51:19.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:51:19.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:51:19.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:51:19.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:51:19.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:51:19.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:51:19.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:51:19.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:51:19.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:51:19.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:51:19.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:51:19.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:51:19.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:51:19.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:51:19.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:51:19.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:51:19.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:51:19.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:51:19.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:51:19.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:51:19.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:51:19.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:51:19.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:51:19.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:51:19.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:51:19.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:51:19.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:51:19.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:51:19.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:51:19.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:51:19.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:51:19.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:51:19.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:51:19.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:51:19.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:51:19.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:51:19.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:51:19.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: main() 2024-01-13T11:51:19.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:51:19.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:51:19.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:51:19.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:51:19.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:51:19.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:51:19.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:51:19.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:51:19.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:51:19.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:51:19.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:51:19.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:51:19.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:51:19.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:51:19.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp4a1qi124:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmph81obn55:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:51:19.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: cluster 2024-01-13T11:51:17.332161+0000 mgr.smithi088.xwlxdh (mgr.14182) 142 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:19.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: audit 2024-01-13T11:51:18.103815+0000 mon.smithi088 (mon.0) 370 : audit [INF] from='client.? 172.21.15.88:0/141449381' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:51:19.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: audit 2024-01-13T11:51:18.111267+0000 mon.smithi088 (mon.0) 371 : audit [INF] from='client.? 172.21.15.88:0/141449381' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:51:19.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:18 smithi138 bash[17620]: cluster 2024-01-13T11:51:18.111310+0000 mon.smithi088 (mon.0) 372 : cluster [DBG] osdmap e21: 0 total, 0 up, 0 in 2024-01-13T11:51:19.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:51:19.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:51:19.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:51:19.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:51:19.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:51:19.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:51:19.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:51:19.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:51:19.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:51:19.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:51:19.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:51:19.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:51:19.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:51:19.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:51:19.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:51:19.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:51:19.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:51:19.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:51:19.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:51:19.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:51:19.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:51:19.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:51:19.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:51:19.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:51:19.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:51:19.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:51:19.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:51:19.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:51:19.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:51:19.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:51:19.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:51:19.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:51:19.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:51:19.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:51:19.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:51:19.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:51:19.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:51:19.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:51:19.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:51:19.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:51:19.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:51:19.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:51:19.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:51:19.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:51:19.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:51:19.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:51:19.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:51:19.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:51:19.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:51:19.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:51:19.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:51:19.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:51:19.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:51:19.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:51:19.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:51:19.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:51:19.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:51:19.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:51:19.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: main() 2024-01-13T11:51:19.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:51:19.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:51:19.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:51:19.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:51:19.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:51:19.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:51:19.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:51:19.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:51:19.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:51:19.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:51:19.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:51:19.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:51:19.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:51:19.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:51:19.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp4a1qi124:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmph81obn55:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:51:19.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: cluster 2024-01-13T11:51:17.332161+0000 mgr.smithi088.xwlxdh (mgr.14182) 142 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:19.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: audit 2024-01-13T11:51:18.103815+0000 mon.smithi088 (mon.0) 370 : audit [INF] from='client.? 172.21.15.88:0/141449381' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:51:19.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: audit 2024-01-13T11:51:18.111267+0000 mon.smithi088 (mon.0) 371 : audit [INF] from='client.? 172.21.15.88:0/141449381' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:51:19.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:18 smithi088 bash[14069]: cluster 2024-01-13T11:51:18.111310+0000 mon.smithi088 (mon.0) 372 : cluster [DBG] osdmap e21: 0 total, 0 up, 0 in 2024-01-13T11:51:20.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:19 smithi138 bash[17620]: cluster 2024-01-13T11:51:19.888984+0000 mon.smithi088 (mon.0) 373 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T11:51:20.379 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:19 smithi088 bash[14069]: cluster 2024-01-13T11:51:19.888984+0000 mon.smithi088 (mon.0) 373 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T11:51:20.793 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:51:21.122 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: cluster 2024-01-13T11:51:19.332523+0000 mgr.smithi088.xwlxdh (mgr.14182) 143 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:21.123 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: cephadm 2024-01-13T11:51:19.533875+0000 mgr.smithi088.xwlxdh (mgr.14182) 144 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:51:21.123 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:51:21.123 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:51:21.123 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: return f(*arg) 2024-01-13T11:51:21.123 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:51:21.123 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:51:21.123 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:51:21.123 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: code, '\n'.join(err))) 2024-01-13T11:51:21.123 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi088/config 2024-01-13T11:51:21.123 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp2trvtxyu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp97g6uir6:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:51:21.123 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:51:21.123 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:51:21.124 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:51:21.124 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new b5463224-51fb-4da9-8869-52eae92055e5 2024-01-13T11:51:21.124 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:51:21.124 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:51:21.124 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:51:21.124 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:51:21.124 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:51:21.124 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:51:21.125 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:51:21.125 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:51:21.125 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:51:21.125 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:51:21.125 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:51:21.125 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:51:21.125 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:51:21.125 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:51:21.126 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:51:21.126 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:51:21.126 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:51:21.126 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:51:21.126 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:51:21.126 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:51:21.126 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:51:21.126 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:51:21.126 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:51:21.126 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:51:21.126 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:51:21.126 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:51:21.126 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:51:21.126 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:51:21.126 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:51:21.127 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:51:21.127 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:51:21.127 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:51:21.127 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:51:21.127 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:51:21.127 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:51:21.127 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:51:21.127 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:51:21.127 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:51:21.127 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:51:21.127 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:51:21.127 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:51:21.127 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:51:21.127 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:51:21.127 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:51:21.128 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:51:21.128 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:51:21.128 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:51:21.128 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:51:21.128 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:51:21.128 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:51:21.128 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:51:21.128 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:51:21.128 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:51:21.128 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:51:21.128 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:51:21.128 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:51:21.128 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:51:21.128 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:51:21.129 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: main() 2024-01-13T11:51:21.129 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:51:21.129 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:51:21.129 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:51:21.129 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:51:21.129 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:51:21.129 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:51:21.129 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:51:21.131 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:51:21.131 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:51:21.131 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:51:21.131 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:51:21.131 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:51:21.131 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:51:21.131 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:51:21.131 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp2trvtxyu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp97g6uir6:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:51:21.131 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: cephadm 2024-01-13T11:51:19.535961+0000 mgr.smithi088.xwlxdh (mgr.14182) 145 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T11:51:21.131 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: service_id: all-available-devices 2024-01-13T11:51:21.131 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: service_name: osd.all-available-devices 2024-01-13T11:51:21.131 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: placement: 2024-01-13T11:51:21.131 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: host_pattern: '*' 2024-01-13T11:51:21.131 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: spec: 2024-01-13T11:51:21.132 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: data_devices: 2024-01-13T11:51:21.132 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: all: true 2024-01-13T11:51:21.132 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: filter_logic: AND 2024-01-13T11:51:21.132 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: objectstore: bluestore 2024-01-13T11:51:21.132 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:51:21.132 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp4a1qi124:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmph81obn55:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:51:21.132 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:51:21.132 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:51:21.132 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:51:21.132 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4c0937fb-b80d-40a7-87f1-1a65720cab2a 2024-01-13T11:51:21.132 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:51:21.132 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:51:21.132 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:51:21.133 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:51:21.133 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:51:21.133 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:51:21.133 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:51:21.133 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:51:21.133 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:51:21.133 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:51:21.133 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:51:21.133 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:51:21.133 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:51:21.133 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:51:21.133 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:51:21.133 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:51:21.133 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:51:21.134 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:51:21.134 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:51:21.134 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:51:21.134 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:51:21.134 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:51:21.134 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:51:21.134 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:51:21.134 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:51:21.134 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:51:21.134 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:51:21.134 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:51:21.134 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:51:21.134 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:51:21.134 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:51:21.134 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:51:21.135 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:51:21.135 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:51:21.136 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:51:21.136 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:51:21.136 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:51:21.136 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:51:21.136 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:51:21.136 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:51:21.136 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:51:21.136 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:51:21.136 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:51:21.136 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:51:21.136 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:51:21.137 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:51:21.137 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:51:21.137 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:51:21.137 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:51:21.137 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:51:21.137 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:51:21.137 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:51:21.137 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:51:21.137 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:51:21.137 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:51:21.137 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:51:21.137 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:51:21.137 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:51:21.137 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: main() 2024-01-13T11:51:21.137 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:51:21.138 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:51:21.138 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:51:21.138 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:51:21.138 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:51:21.138 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:51:21.138 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:51:21.138 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:51:21.138 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:51:21.138 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:51:21.138 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:51:21.138 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:51:21.138 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:51:21.138 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:51:21.138 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp4a1qi124:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmph81obn55:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:51:21.139 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:51:21.139 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T11:51:21.139 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: if self._apply_service(spec): 2024-01-13T11:51:21.139 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T11:51:21.139 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T11:51:21.139 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T11:51:21.139 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T11:51:21.139 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T11:51:21.139 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T11:51:21.139 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T11:51:21.139 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T11:51:21.139 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T11:51:21.139 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: raise self._value 2024-01-13T11:51:21.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T11:51:21.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: result = (True, func(*args, **kwds)) 2024-01-13T11:51:21.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T11:51:21.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: return list(map(*args)) 2024-01-13T11:51:21.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:51:21.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: return f(*arg) 2024-01-13T11:51:21.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:51:21.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:51:21.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:51:21.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: code, '\n'.join(err))) 2024-01-13T11:51:21.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:51:21.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp4a1qi124:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmph81obn55:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:51:21.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:51:21.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:51:21.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:51:21.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4c0937fb-b80d-40a7-87f1-1a65720cab2a 2024-01-13T11:51:21.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:51:21.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:51:21.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:51:21.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:51:21.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:51:21.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:51:21.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:51:21.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:51:21.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:51:21.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:51:21.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:51:21.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:51:21.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:51:21.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:51:21.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:51:21.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:51:21.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:51:21.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:51:21.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:51:21.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:51:21.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:51:21.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:51:21.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:51:21.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:51:21.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:51:21.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:51:21.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:51:21.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:51:21.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:51:21.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:51:21.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:51:21.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:51:21.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:51:21.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:51:21.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:51:21.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:51:21.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:51:21.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:51:21.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:51:21.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:51:21.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:51:21.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:51:21.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:51:21.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:51:21.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:51:21.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:51:21.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:51:21.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:51:21.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:51:21.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:51:21.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:51:21.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:51:21.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:51:21.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:51:21.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:51:21.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:51:21.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:51:21.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:51:21.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: main() 2024-01-13T11:51:21.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:51:21.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:51:21.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:51:21.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:51:21.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:51:21.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:51:21.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:51:21.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:51:21.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:51:21.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:51:21.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:51:21.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:51:21.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:51:21.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:51:21.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp4a1qi124:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmph81obn55:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:51:21.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: cluster 2024-01-13T11:51:19.536840+0000 mgr.smithi088.xwlxdh (mgr.14182) 146 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:21.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:20 smithi088 bash[14069]: audit 2024-01-13T11:51:20.803966+0000 mon.smithi088 (mon.0) 374 : audit [DBG] from='client.? 172.21.15.88:0/1982236853' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:51:21.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: cluster 2024-01-13T11:51:19.332523+0000 mgr.smithi088.xwlxdh (mgr.14182) 143 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:21.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: cephadm 2024-01-13T11:51:19.533875+0000 mgr.smithi088.xwlxdh (mgr.14182) 144 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:51:21.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:51:21.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:51:21.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: return f(*arg) 2024-01-13T11:51:21.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:51:21.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:51:21.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:51:21.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: code, '\n'.join(err))) 2024-01-13T11:51:21.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi088/config 2024-01-13T11:51:21.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp2trvtxyu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp97g6uir6:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:51:21.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:51:21.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:51:21.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:51:21.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new b5463224-51fb-4da9-8869-52eae92055e5 2024-01-13T11:51:21.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:51:21.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:51:21.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:51:21.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:51:21.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:51:21.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:51:21.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:51:21.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:51:21.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:51:21.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:51:21.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:51:21.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:51:21.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:51:21.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:51:21.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:51:21.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:51:21.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:51:21.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:51:21.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:51:21.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:51:21.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:51:21.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:51:21.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:51:21.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:51:21.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:51:21.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:51:21.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:51:21.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:51:21.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:51:21.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:51:21.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:51:21.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:51:21.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:51:21.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:51:21.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:51:21.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:51:21.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:51:21.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:51:21.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:51:21.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:51:21.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:51:21.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:51:21.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:51:21.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:51:21.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:51:21.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:51:21.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:51:21.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:51:21.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:51:21.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:51:21.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:51:21.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:51:21.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:51:21.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:51:21.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:51:21.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:51:21.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:51:21.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:51:21.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: main() 2024-01-13T11:51:21.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:51:21.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:51:21.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:51:21.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:51:21.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:51:21.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:51:21.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:51:21.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:51:21.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:51:21.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:51:21.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:51:21.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:51:21.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:51:21.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:51:21.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp2trvtxyu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp97g6uir6:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:51:21.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: cephadm 2024-01-13T11:51:19.535961+0000 mgr.smithi088.xwlxdh (mgr.14182) 145 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T11:51:21.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: service_id: all-available-devices 2024-01-13T11:51:21.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: service_name: osd.all-available-devices 2024-01-13T11:51:21.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: placement: 2024-01-13T11:51:21.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: host_pattern: '*' 2024-01-13T11:51:21.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: spec: 2024-01-13T11:51:21.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: data_devices: 2024-01-13T11:51:21.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: all: true 2024-01-13T11:51:21.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: filter_logic: AND 2024-01-13T11:51:21.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: objectstore: bluestore 2024-01-13T11:51:21.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:51:21.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp4a1qi124:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmph81obn55:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:51:21.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:51:21.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:51:21.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:51:21.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4c0937fb-b80d-40a7-87f1-1a65720cab2a 2024-01-13T11:51:21.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:51:21.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:51:21.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:51:21.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:51:21.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:51:21.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:51:21.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:51:21.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:51:21.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:51:21.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:51:21.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:51:21.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:51:21.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:51:21.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:51:21.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:51:21.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:51:21.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:51:21.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:51:21.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:51:21.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:51:21.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:51:21.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:51:21.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:51:21.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:51:21.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:51:21.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:51:21.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:51:21.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:51:21.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:51:21.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:51:21.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:51:21.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:51:21.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:51:21.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:51:21.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:51:21.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:51:21.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:51:21.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:51:21.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:51:21.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:51:21.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:51:21.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:51:21.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:51:21.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:51:21.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:51:21.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:51:21.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:51:21.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:51:21.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:51:21.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:51:21.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:51:21.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:51:21.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:51:21.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:51:21.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:51:21.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:51:21.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:51:21.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:51:21.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: main() 2024-01-13T11:51:21.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:51:21.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:51:21.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:51:21.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:51:21.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:51:21.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:51:21.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:51:21.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:51:21.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:51:21.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:51:21.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:51:21.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:51:21.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:51:21.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:51:21.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp4a1qi124:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmph81obn55:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:51:21.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:51:21.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T11:51:21.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: if self._apply_service(spec): 2024-01-13T11:51:21.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T11:51:21.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T11:51:21.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T11:51:21.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T11:51:21.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T11:51:21.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T11:51:21.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T11:51:21.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T11:51:21.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T11:51:21.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: raise self._value 2024-01-13T11:51:21.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T11:51:21.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: result = (True, func(*args, **kwds)) 2024-01-13T11:51:21.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T11:51:21.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: return list(map(*args)) 2024-01-13T11:51:21.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:51:21.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: return f(*arg) 2024-01-13T11:51:21.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:51:21.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:51:21.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:51:21.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: code, '\n'.join(err))) 2024-01-13T11:51:21.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:51:21.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp4a1qi124:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmph81obn55:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:51:21.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:51:21.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:51:21.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:51:21.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4c0937fb-b80d-40a7-87f1-1a65720cab2a 2024-01-13T11:51:21.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:51:21.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:51:21.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:51:21.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:51:21.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:51:21.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:51:21.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:51:21.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:51:21.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:51:21.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:51:21.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:51:21.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:51:21.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:51:21.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:51:21.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:51:21.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:51:21.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:51:21.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:51:21.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:51:21.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:51:21.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:51:21.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:51:21.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:51:21.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:51:21.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:51:21.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:51:21.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:51:21.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:51:21.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:51:21.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:51:21.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:51:21.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:51:21.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:51:21.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:51:21.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:51:21.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:51:21.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:51:21.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:51:21.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:51:21.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:51:21.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:51:21.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:51:21.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:51:21.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:51:21.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:51:21.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:51:21.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:51:21.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:51:21.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:51:21.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:51:21.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:51:21.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:51:21.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:51:21.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:51:21.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:51:21.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:51:21.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:51:21.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:51:21.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: main() 2024-01-13T11:51:21.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:51:21.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:51:21.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:51:21.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:51:21.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:51:21.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:51:21.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:51:21.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:51:21.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:51:21.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:51:21.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:51:21.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:51:21.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:51:21.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:51:21.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp4a1qi124:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmph81obn55:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:51:21.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: cluster 2024-01-13T11:51:19.536840+0000 mgr.smithi088.xwlxdh (mgr.14182) 146 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:21.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:20 smithi138 bash[17620]: audit 2024-01-13T11:51:20.803966+0000 mon.smithi088 (mon.0) 374 : audit [DBG] from='client.? 172.21.15.88:0/1982236853' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:51:21.564 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:51:22.565 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:51:23.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:22 smithi088 bash[14069]: cluster 2024-01-13T11:51:21.537343+0000 mgr.smithi088.xwlxdh (mgr.14182) 147 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:23.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:22 smithi138 bash[17620]: cluster 2024-01-13T11:51:21.537343+0000 mgr.smithi088.xwlxdh (mgr.14182) 147 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:24.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:24 smithi138 bash[17620]: cluster 2024-01-13T11:51:23.537953+0000 mgr.smithi088.xwlxdh (mgr.14182) 148 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:24.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:24 smithi088 bash[14069]: cluster 2024-01-13T11:51:23.537953+0000 mgr.smithi088.xwlxdh (mgr.14182) 148 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:25.606 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:51:26.340 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:51:26.641 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:26 smithi088 bash[14069]: cluster 2024-01-13T11:51:25.538506+0000 mgr.smithi088.xwlxdh (mgr.14182) 149 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:26.641 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:26 smithi088 bash[14069]: audit 2024-01-13T11:51:25.616878+0000 mon.smithi088 (mon.0) 375 : audit [DBG] from='client.? 172.21.15.88:0/1072320733' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:51:26.875 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:26 smithi138 bash[17620]: cluster 2024-01-13T11:51:25.538506+0000 mgr.smithi088.xwlxdh (mgr.14182) 149 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:26.875 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:26 smithi138 bash[17620]: audit 2024-01-13T11:51:25.616878+0000 mon.smithi088 (mon.0) 375 : audit [DBG] from='client.? 172.21.15.88:0/1072320733' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:51:27.341 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:51:29.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:28 smithi138 bash[17620]: cluster 2024-01-13T11:51:27.539435+0000 mgr.smithi088.xwlxdh (mgr.14182) 150 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:29.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:28 smithi088 bash[14069]: cluster 2024-01-13T11:51:27.539435+0000 mgr.smithi088.xwlxdh (mgr.14182) 150 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:30.324 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:51:31.114 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:30 smithi088 bash[14069]: cluster 2024-01-13T11:51:29.540523+0000 mgr.smithi088.xwlxdh (mgr.14182) 151 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:31.114 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:30 smithi088 bash[14069]: audit 2024-01-13T11:51:29.730311+0000 mon.smithi088 (mon.0) 376 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:51:31.114 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:30 smithi088 bash[14069]: audit 2024-01-13T11:51:30.334934+0000 mon.smithi088 (mon.0) 377 : audit [DBG] from='client.? 172.21.15.88:0/1201335329' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:51:31.115 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:51:31.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:30 smithi138 bash[17620]: cluster 2024-01-13T11:51:29.540523+0000 mgr.smithi088.xwlxdh (mgr.14182) 151 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:31.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:30 smithi138 bash[17620]: audit 2024-01-13T11:51:29.730311+0000 mon.smithi088 (mon.0) 376 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:51:31.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:30 smithi138 bash[17620]: audit 2024-01-13T11:51:30.334934+0000 mon.smithi088 (mon.0) 377 : audit [DBG] from='client.? 172.21.15.88:0/1201335329' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:51:32.116 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:51:32.874 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:32 smithi138 bash[17620]: cluster 2024-01-13T11:51:31.541526+0000 mgr.smithi088.xwlxdh (mgr.14182) 152 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:33.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:32 smithi088 bash[14069]: cluster 2024-01-13T11:51:31.541526+0000 mgr.smithi088.xwlxdh (mgr.14182) 152 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:35.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:34 smithi138 bash[17620]: cluster 2024-01-13T11:51:33.542081+0000 mgr.smithi088.xwlxdh (mgr.14182) 153 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:35.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:34 smithi088 bash[14069]: cluster 2024-01-13T11:51:33.542081+0000 mgr.smithi088.xwlxdh (mgr.14182) 153 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:35.269 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:51:36.042 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:51:36.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:35 smithi088 bash[14069]: audit 2024-01-13T11:51:35.280279+0000 mon.smithi088 (mon.0) 378 : audit [DBG] from='client.? 172.21.15.88:0/2022008675' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:51:36.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:35 smithi138 bash[17620]: audit 2024-01-13T11:51:35.280279+0000 mon.smithi088 (mon.0) 378 : audit [DBG] from='client.? 172.21.15.88:0/2022008675' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:51:37.043 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:51:37.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:36 smithi088 bash[14069]: cluster 2024-01-13T11:51:35.542955+0000 mgr.smithi088.xwlxdh (mgr.14182) 154 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:37.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:36 smithi138 bash[17620]: cluster 2024-01-13T11:51:35.542955+0000 mgr.smithi088.xwlxdh (mgr.14182) 154 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:39.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:38 smithi088 bash[14069]: cluster 2024-01-13T11:51:37.543983+0000 mgr.smithi088.xwlxdh (mgr.14182) 155 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:39.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:38 smithi138 bash[17620]: cluster 2024-01-13T11:51:37.543983+0000 mgr.smithi088.xwlxdh (mgr.14182) 155 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:40.054 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:51:40.801 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:51:41.121 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:40 smithi088 bash[14069]: cluster 2024-01-13T11:51:39.544992+0000 mgr.smithi088.xwlxdh (mgr.14182) 156 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:41.121 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:40 smithi088 bash[14069]: audit 2024-01-13T11:51:40.064920+0000 mon.smithi088 (mon.0) 379 : audit [DBG] from='client.? 172.21.15.88:0/2058454272' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:51:41.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:40 smithi138 bash[17620]: cluster 2024-01-13T11:51:39.544992+0000 mgr.smithi088.xwlxdh (mgr.14182) 156 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:41.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:40 smithi138 bash[17620]: audit 2024-01-13T11:51:40.064920+0000 mon.smithi088 (mon.0) 379 : audit [DBG] from='client.? 172.21.15.88:0/2058454272' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:51:41.802 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:51:43.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:42 smithi088 bash[14069]: cluster 2024-01-13T11:51:41.545965+0000 mgr.smithi088.xwlxdh (mgr.14182) 157 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:43.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:42 smithi138 bash[17620]: cluster 2024-01-13T11:51:41.545965+0000 mgr.smithi088.xwlxdh (mgr.14182) 157 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:45.026 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:51:45.278 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:44 smithi088 bash[14069]: cluster 2024-01-13T11:51:43.546994+0000 mgr.smithi088.xwlxdh (mgr.14182) 158 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:45.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:44 smithi138 bash[17620]: cluster 2024-01-13T11:51:43.546994+0000 mgr.smithi088.xwlxdh (mgr.14182) 158 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:45.768 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:51:46.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:45 smithi088 bash[14069]: audit 2024-01-13T11:51:45.036433+0000 mon.smithi088 (mon.0) 380 : audit [DBG] from='client.? 172.21.15.88:0/1952454871' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:51:46.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:45 smithi138 bash[17620]: audit 2024-01-13T11:51:45.036433+0000 mon.smithi088 (mon.0) 380 : audit [DBG] from='client.? 172.21.15.88:0/1952454871' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:51:46.769 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:51:47.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:46 smithi088 bash[14069]: cluster 2024-01-13T11:51:45.548010+0000 mgr.smithi088.xwlxdh (mgr.14182) 159 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:47.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:46 smithi138 bash[17620]: cluster 2024-01-13T11:51:45.548010+0000 mgr.smithi088.xwlxdh (mgr.14182) 159 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:49.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:48 smithi088 bash[14069]: cluster 2024-01-13T11:51:47.548511+0000 mgr.smithi088.xwlxdh (mgr.14182) 160 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:49.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:48 smithi138 bash[17620]: cluster 2024-01-13T11:51:47.548511+0000 mgr.smithi088.xwlxdh (mgr.14182) 160 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:50.047 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:51:50.866 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:51:51.121 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:50 smithi088 bash[14069]: cluster 2024-01-13T11:51:49.549444+0000 mgr.smithi088.xwlxdh (mgr.14182) 161 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:51.121 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:50 smithi088 bash[14069]: audit 2024-01-13T11:51:50.057951+0000 mon.smithi088 (mon.0) 381 : audit [DBG] from='client.? 172.21.15.88:0/1619889031' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:51:51.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:50 smithi138 bash[17620]: cluster 2024-01-13T11:51:49.549444+0000 mgr.smithi088.xwlxdh (mgr.14182) 161 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:51.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:50 smithi138 bash[17620]: audit 2024-01-13T11:51:50.057951+0000 mon.smithi088 (mon.0) 381 : audit [DBG] from='client.? 172.21.15.88:0/1619889031' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:51:51.867 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:51:53.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:52 smithi088 bash[14069]: cluster 2024-01-13T11:51:51.550392+0000 mgr.smithi088.xwlxdh (mgr.14182) 162 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:53.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:52 smithi138 bash[17620]: cluster 2024-01-13T11:51:51.550392+0000 mgr.smithi088.xwlxdh (mgr.14182) 162 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:55.007 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:51:55.293 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:54 smithi088 bash[14069]: cluster 2024-01-13T11:51:53.551426+0000 mgr.smithi088.xwlxdh (mgr.14182) 163 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:55.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:54 smithi138 bash[17620]: cluster 2024-01-13T11:51:53.551426+0000 mgr.smithi088.xwlxdh (mgr.14182) 163 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:55.787 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:51:56.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:55 smithi088 bash[14069]: audit 2024-01-13T11:51:55.017787+0000 mon.smithi088 (mon.0) 382 : audit [DBG] from='client.? 172.21.15.88:0/3021085612' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:51:56.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:55 smithi088 bash[14069]: cluster 2024-01-13T11:51:55.552446+0000 mgr.smithi088.xwlxdh (mgr.14182) 164 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:56.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:55 smithi138 bash[17620]: audit 2024-01-13T11:51:55.017787+0000 mon.smithi088 (mon.0) 382 : audit [DBG] from='client.? 172.21.15.88:0/3021085612' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:51:56.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:55 smithi138 bash[17620]: cluster 2024-01-13T11:51:55.552446+0000 mgr.smithi088.xwlxdh (mgr.14182) 164 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:56.788 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:51:59.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:58 smithi088 bash[14069]: cluster 2024-01-13T11:51:57.553337+0000 mgr.smithi088.xwlxdh (mgr.14182) 165 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:59.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:58 smithi138 bash[17620]: cluster 2024-01-13T11:51:57.553337+0000 mgr.smithi088.xwlxdh (mgr.14182) 165 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:51:59.768 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:52:00.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:51:59 smithi088 bash[14069]: audit 2024-01-13T11:51:59.778708+0000 mon.smithi088 (mon.0) 383 : audit [DBG] from='client.? 172.21.15.88:0/3120565444' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:52:00.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:51:59 smithi138 bash[17620]: audit 2024-01-13T11:51:59.778708+0000 mon.smithi088 (mon.0) 383 : audit [DBG] from='client.? 172.21.15.88:0/3120565444' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:52:00.466 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:52:01.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:00 smithi088 bash[14069]: cluster 2024-01-13T11:51:59.554087+0000 mgr.smithi088.xwlxdh (mgr.14182) 166 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:01.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:00 smithi138 bash[17620]: cluster 2024-01-13T11:51:59.554087+0000 mgr.smithi088.xwlxdh (mgr.14182) 166 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:01.468 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:52:03.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:02 smithi138 bash[17620]: cluster 2024-01-13T11:52:01.555096+0000 mgr.smithi088.xwlxdh (mgr.14182) 167 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:03.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:02 smithi088 bash[14069]: cluster 2024-01-13T11:52:01.555096+0000 mgr.smithi088.xwlxdh (mgr.14182) 167 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:04.614 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:52:05.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:04 smithi088 bash[14069]: cluster 2024-01-13T11:52:03.556084+0000 mgr.smithi088.xwlxdh (mgr.14182) 168 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:05.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:04 smithi088 bash[14069]: audit 2024-01-13T11:52:04.625111+0000 mon.smithi088 (mon.0) 384 : audit [DBG] from='client.? 172.21.15.88:0/4038171597' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:52:05.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:04 smithi138 bash[17620]: cluster 2024-01-13T11:52:03.556084+0000 mgr.smithi088.xwlxdh (mgr.14182) 168 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:05.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:04 smithi138 bash[17620]: audit 2024-01-13T11:52:04.625111+0000 mon.smithi088 (mon.0) 384 : audit [DBG] from='client.? 172.21.15.88:0/4038171597' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:52:05.383 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:52:06.384 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:52:07.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:06 smithi138 bash[17620]: cluster 2024-01-13T11:52:05.557125+0000 mgr.smithi088.xwlxdh (mgr.14182) 169 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:07.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:06 smithi088 bash[14069]: cluster 2024-01-13T11:52:05.557125+0000 mgr.smithi088.xwlxdh (mgr.14182) 169 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:09.207 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:08 smithi088 bash[14069]: cluster 2024-01-13T11:52:07.557676+0000 mgr.smithi088.xwlxdh (mgr.14182) 170 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:09.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:08 smithi138 bash[17620]: cluster 2024-01-13T11:52:07.557676+0000 mgr.smithi088.xwlxdh (mgr.14182) 170 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:09.609 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:52:10.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:09 smithi088 bash[14069]: audit 2024-01-13T11:52:09.620626+0000 mon.smithi088 (mon.0) 385 : audit [DBG] from='client.? 172.21.15.88:0/957688874' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:52:10.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:09 smithi138 bash[17620]: audit 2024-01-13T11:52:09.620626+0000 mon.smithi088 (mon.0) 385 : audit [DBG] from='client.? 172.21.15.88:0/957688874' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:52:10.427 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:52:11.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:10 smithi138 bash[17620]: cluster 2024-01-13T11:52:09.558678+0000 mgr.smithi088.xwlxdh (mgr.14182) 171 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:11.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:10 smithi088 bash[14069]: cluster 2024-01-13T11:52:09.558678+0000 mgr.smithi088.xwlxdh (mgr.14182) 171 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:11.429 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:52:13.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:12 smithi088 bash[14069]: cluster 2024-01-13T11:52:11.559315+0000 mgr.smithi088.xwlxdh (mgr.14182) 172 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:13.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:12 smithi138 bash[17620]: cluster 2024-01-13T11:52:11.559315+0000 mgr.smithi088.xwlxdh (mgr.14182) 172 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:14.524 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:52:15.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:14 smithi088 bash[14069]: cluster 2024-01-13T11:52:13.560237+0000 mgr.smithi088.xwlxdh (mgr.14182) 173 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:15.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:14 smithi088 bash[14069]: audit 2024-01-13T11:52:14.534669+0000 mon.smithi088 (mon.0) 386 : audit [DBG] from='client.? 172.21.15.88:0/2627252867' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:52:15.338 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:52:15.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:14 smithi138 bash[17620]: cluster 2024-01-13T11:52:13.560237+0000 mgr.smithi088.xwlxdh (mgr.14182) 173 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:15.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:14 smithi138 bash[17620]: audit 2024-01-13T11:52:14.534669+0000 mon.smithi088 (mon.0) 386 : audit [DBG] from='client.? 172.21.15.88:0/2627252867' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:52:16.340 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:52:17.372 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:16 smithi088 bash[14069]: cluster 2024-01-13T11:52:15.561245+0000 mgr.smithi088.xwlxdh (mgr.14182) 174 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:17.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:16 smithi138 bash[17620]: cluster 2024-01-13T11:52:15.561245+0000 mgr.smithi088.xwlxdh (mgr.14182) 174 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:19.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:18 smithi138 bash[17620]: cluster 2024-01-13T11:52:17.562248+0000 mgr.smithi088.xwlxdh (mgr.14182) 175 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:19.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:18 smithi088 bash[14069]: cluster 2024-01-13T11:52:17.562248+0000 mgr.smithi088.xwlxdh (mgr.14182) 175 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:19.394 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:52:20.148 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:52:20.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:19 smithi138 bash[17620]: audit 2024-01-13T11:52:19.405098+0000 mon.smithi088 (mon.0) 387 : audit [DBG] from='client.? 172.21.15.88:0/2132531400' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:52:20.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:19 smithi138 bash[17620]: audit 2024-01-13T11:52:19.538962+0000 mon.smithi088 (mon.0) 388 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:52:20.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:19 smithi138 bash[17620]: cluster 2024-01-13T11:52:19.562747+0000 mgr.smithi088.xwlxdh (mgr.14182) 176 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:20.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:19 smithi088 bash[14069]: audit 2024-01-13T11:52:19.405098+0000 mon.smithi088 (mon.0) 387 : audit [DBG] from='client.? 172.21.15.88:0/2132531400' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:52:20.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:19 smithi088 bash[14069]: audit 2024-01-13T11:52:19.538962+0000 mon.smithi088 (mon.0) 388 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:52:20.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:19 smithi088 bash[14069]: cluster 2024-01-13T11:52:19.562747+0000 mgr.smithi088.xwlxdh (mgr.14182) 176 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:21.150 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:52:22.873 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:22 smithi138 bash[17620]: cluster 2024-01-13T11:52:21.563714+0000 mgr.smithi088.xwlxdh (mgr.14182) 177 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:22.891 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:22 smithi088 bash[14069]: cluster 2024-01-13T11:52:21.563714+0000 mgr.smithi088.xwlxdh (mgr.14182) 177 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:24.203 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:52:24.939 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:52:25.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:24 smithi088 bash[14069]: cluster 2024-01-13T11:52:23.564628+0000 mgr.smithi088.xwlxdh (mgr.14182) 178 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:25.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:24 smithi088 bash[14069]: audit 2024-01-13T11:52:24.213884+0000 mon.smithi088 (mon.0) 389 : audit [DBG] from='client.? 172.21.15.88:0/3190940433' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:52:25.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:24 smithi138 bash[17620]: cluster 2024-01-13T11:52:23.564628+0000 mgr.smithi088.xwlxdh (mgr.14182) 178 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:25.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:24 smithi138 bash[17620]: audit 2024-01-13T11:52:24.213884+0000 mon.smithi088 (mon.0) 389 : audit [DBG] from='client.? 172.21.15.88:0/3190940433' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:52:25.941 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:52:26.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:26 smithi138 bash[17620]: audit 2024-01-13T11:52:25.024848+0000 mon.smithi088 (mon.0) 390 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:52:26.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:26 smithi138 bash[17620]: audit 2024-01-13T11:52:25.299783+0000 mon.smithi088 (mon.0) 391 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:52:26.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:26 smithi138 bash[17620]: audit 2024-01-13T11:52:25.301700+0000 mon.smithi088 (mon.0) 392 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T11:52:26.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:26 smithi138 bash[17620]: audit 2024-01-13T11:52:25.310462+0000 mon.smithi088 (mon.0) 393 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:52:26.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:26 smithi138 bash[17620]: cluster 2024-01-13T11:52:25.311167+0000 mgr.smithi088.xwlxdh (mgr.14182) 179 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:26.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:26 smithi138 bash[17620]: audit 2024-01-13T11:52:25.319167+0000 mon.smithi088 (mon.0) 394 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T11:52:26.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:26 smithi138 bash[17620]: audit 2024-01-13T11:52:25.325321+0000 mon.smithi088 (mon.0) 395 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:52:26.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:26 smithi138 bash[17620]: audit 2024-01-13T11:52:25.328614+0000 mon.smithi088 (mon.0) 396 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:52:26.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:26 smithi138 bash[17620]: audit 2024-01-13T11:52:25.329134+0000 mon.smithi088 (mon.0) 397 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:52:26.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:26 smithi138 bash[17620]: audit 2024-01-13T11:52:25.330155+0000 mon.smithi088 (mon.0) 398 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:52:26.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:26 smithi088 bash[14069]: audit 2024-01-13T11:52:25.024848+0000 mon.smithi088 (mon.0) 390 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:52:26.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:26 smithi088 bash[14069]: audit 2024-01-13T11:52:25.299783+0000 mon.smithi088 (mon.0) 391 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:52:26.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:26 smithi088 bash[14069]: audit 2024-01-13T11:52:25.301700+0000 mon.smithi088 (mon.0) 392 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T11:52:26.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:26 smithi088 bash[14069]: audit 2024-01-13T11:52:25.310462+0000 mon.smithi088 (mon.0) 393 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:52:26.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:26 smithi088 bash[14069]: cluster 2024-01-13T11:52:25.311167+0000 mgr.smithi088.xwlxdh (mgr.14182) 179 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:26.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:26 smithi088 bash[14069]: audit 2024-01-13T11:52:25.319167+0000 mon.smithi088 (mon.0) 394 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T11:52:26.392 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:26 smithi088 bash[14069]: audit 2024-01-13T11:52:25.325321+0000 mon.smithi088 (mon.0) 395 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:52:26.392 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:26 smithi088 bash[14069]: audit 2024-01-13T11:52:25.328614+0000 mon.smithi088 (mon.0) 396 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:52:26.392 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:26 smithi088 bash[14069]: audit 2024-01-13T11:52:25.329134+0000 mon.smithi088 (mon.0) 397 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:52:26.392 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:26 smithi088 bash[14069]: audit 2024-01-13T11:52:25.330155+0000 mon.smithi088 (mon.0) 398 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:52:27.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:27 smithi138 bash[17620]: cluster 2024-01-13T11:52:26.308019+0000 mon.smithi088 (mon.0) 399 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T11:52:27.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:27 smithi138 bash[17620]: cluster 2024-01-13T11:52:26.308063+0000 mon.smithi088 (mon.0) 400 : cluster [INF] Cluster is now healthy 2024-01-13T11:52:27.378 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:27 smithi088 bash[14069]: cluster 2024-01-13T11:52:26.308019+0000 mon.smithi088 (mon.0) 399 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T11:52:27.378 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:27 smithi088 bash[14069]: cluster 2024-01-13T11:52:26.308063+0000 mon.smithi088 (mon.0) 400 : cluster [INF] Cluster is now healthy 2024-01-13T11:52:29.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:28 smithi088 bash[14069]: cluster 2024-01-13T11:52:27.311611+0000 mgr.smithi088.xwlxdh (mgr.14182) 180 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:29.210 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:52:29.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:28 smithi138 bash[17620]: cluster 2024-01-13T11:52:27.311611+0000 mgr.smithi088.xwlxdh (mgr.14182) 180 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:29.983 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:52:30.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:29 smithi138 bash[17620]: audit 2024-01-13T11:52:29.220688+0000 mon.smithi088 (mon.0) 401 : audit [DBG] from='client.? 172.21.15.88:0/4066736804' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:52:30.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:29 smithi138 bash[17620]: audit 2024-01-13T11:52:29.731373+0000 mon.smithi088 (mon.0) 402 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:52:30.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:29 smithi088 bash[14069]: audit 2024-01-13T11:52:29.220688+0000 mon.smithi088 (mon.0) 401 : audit [DBG] from='client.? 172.21.15.88:0/4066736804' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:52:30.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:29 smithi088 bash[14069]: audit 2024-01-13T11:52:29.731373+0000 mon.smithi088 (mon.0) 402 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:52:30.984 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:52:31.209 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:30 smithi138 bash[17620]: cluster 2024-01-13T11:52:29.312111+0000 mgr.smithi088.xwlxdh (mgr.14182) 181 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:31.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:30 smithi088 bash[14069]: cluster 2024-01-13T11:52:29.312111+0000 mgr.smithi088.xwlxdh (mgr.14182) 181 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:33.027 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:32 smithi088 bash[14069]: cluster 2024-01-13T11:52:31.312506+0000 mgr.smithi088.xwlxdh (mgr.14182) 182 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:33.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:32 smithi138 bash[17620]: cluster 2024-01-13T11:52:31.312506+0000 mgr.smithi088.xwlxdh (mgr.14182) 182 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:34.911 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:34 smithi088 bash[14069]: cluster 2024-01-13T11:52:33.312933+0000 mgr.smithi088.xwlxdh (mgr.14182) 183 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:35.123 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:34 smithi138 bash[17620]: cluster 2024-01-13T11:52:33.312933+0000 mgr.smithi088.xwlxdh (mgr.14182) 183 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:35.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:34 smithi138 bash[17620]: audit 2024-01-13T11:52:34.039547+0000 mon.smithi138 (mon.1) 10 : audit [INF] from='client.? 172.21.15.138:0/1688157579' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e639c875-7aba-4495-9772-51448672a2bc"}]: dispatch 2024-01-13T11:52:35.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:34 smithi138 bash[17620]: audit 2024-01-13T11:52:34.040017+0000 mon.smithi088 (mon.0) 403 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e639c875-7aba-4495-9772-51448672a2bc"}]: dispatch 2024-01-13T11:52:35.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:34 smithi138 bash[17620]: audit 2024-01-13T11:52:34.046368+0000 mon.smithi088 (mon.0) 404 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e639c875-7aba-4495-9772-51448672a2bc"}]': finished 2024-01-13T11:52:35.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:34 smithi138 bash[17620]: cluster 2024-01-13T11:52:34.046424+0000 mon.smithi088 (mon.0) 405 : cluster [DBG] osdmap e22: 1 total, 0 up, 1 in 2024-01-13T11:52:35.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:34 smithi138 bash[17620]: audit 2024-01-13T11:52:34.046512+0000 mon.smithi088 (mon.0) 406 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:52:35.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:34 smithi138 bash[17620]: audit 2024-01-13T11:52:34.398851+0000 mon.smithi088 (mon.0) 407 : audit [INF] from='client.? 172.21.15.138:0/2590224936' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:52:35.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:34 smithi138 bash[17620]: audit 2024-01-13T11:52:34.406296+0000 mon.smithi088 (mon.0) 408 : audit [INF] from='client.? 172.21.15.138:0/2590224936' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:52:35.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:34 smithi138 bash[17620]: cluster 2024-01-13T11:52:34.406403+0000 mon.smithi088 (mon.0) 409 : cluster [DBG] osdmap e23: 0 total, 0 up, 0 in 2024-01-13T11:52:35.250 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:34 smithi088 bash[14069]: audit 2024-01-13T11:52:34.039547+0000 mon.smithi138 (mon.1) 10 : audit [INF] from='client.? 172.21.15.138:0/1688157579' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e639c875-7aba-4495-9772-51448672a2bc"}]: dispatch 2024-01-13T11:52:35.250 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:34 smithi088 bash[14069]: audit 2024-01-13T11:52:34.040017+0000 mon.smithi088 (mon.0) 403 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e639c875-7aba-4495-9772-51448672a2bc"}]: dispatch 2024-01-13T11:52:35.250 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:34 smithi088 bash[14069]: audit 2024-01-13T11:52:34.046368+0000 mon.smithi088 (mon.0) 404 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e639c875-7aba-4495-9772-51448672a2bc"}]': finished 2024-01-13T11:52:35.250 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:34 smithi088 bash[14069]: cluster 2024-01-13T11:52:34.046424+0000 mon.smithi088 (mon.0) 405 : cluster [DBG] osdmap e22: 1 total, 0 up, 1 in 2024-01-13T11:52:35.250 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:34 smithi088 bash[14069]: audit 2024-01-13T11:52:34.046512+0000 mon.smithi088 (mon.0) 406 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:52:35.250 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:34 smithi088 bash[14069]: audit 2024-01-13T11:52:34.398851+0000 mon.smithi088 (mon.0) 407 : audit [INF] from='client.? 172.21.15.138:0/2590224936' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:52:35.250 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:34 smithi088 bash[14069]: audit 2024-01-13T11:52:34.406296+0000 mon.smithi088 (mon.0) 408 : audit [INF] from='client.? 172.21.15.138:0/2590224936' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:52:35.250 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:34 smithi088 bash[14069]: cluster 2024-01-13T11:52:34.406403+0000 mon.smithi088 (mon.0) 409 : cluster [DBG] osdmap e23: 0 total, 0 up, 0 in 2024-01-13T11:52:35.266 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:52:36.071 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":23,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:52:36.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:35 smithi138 bash[17620]: audit 2024-01-13T11:52:35.277675+0000 mon.smithi088 (mon.0) 410 : audit [DBG] from='client.? 172.21.15.88:0/2001651564' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:52:36.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:35 smithi088 bash[14069]: audit 2024-01-13T11:52:35.277675+0000 mon.smithi088 (mon.0) 410 : audit [DBG] from='client.? 172.21.15.88:0/2001651564' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:52:37.072 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:52:37.094 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: cephadm 2024-01-13T11:52:35.260736+0000 mgr.smithi088.xwlxdh (mgr.14182) 184 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:52:37.094 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:52:37.095 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:52:37.095 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: return f(*arg) 2024-01-13T11:52:37.095 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:52:37.095 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:52:37.095 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:52:37.095 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: code, '\n'.join(err))) 2024-01-13T11:52:37.095 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:52:37.095 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpan_vrkjj:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp4rhux0v3:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:52:37.095 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:52:37.095 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:52:37.095 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:52:37.095 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new e639c875-7aba-4495-9772-51448672a2bc 2024-01-13T11:52:37.095 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:52:37.095 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:52:37.096 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:52:37.096 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:52:37.096 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:52:37.096 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:52:37.096 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:52:37.097 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:52:37.097 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:52:37.097 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:52:37.097 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:52:37.097 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:52:37.097 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:52:37.097 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:52:37.097 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:52:37.097 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:52:37.097 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:52:37.097 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:52:37.097 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:52:37.097 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:52:37.098 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:52:37.098 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:52:37.098 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:52:37.098 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:52:37.098 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:52:37.098 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:52:37.098 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:52:37.098 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:52:37.098 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:52:37.098 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:52:37.098 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:52:37.098 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:52:37.098 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:52:37.098 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:52:37.099 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:52:37.099 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:52:37.099 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:52:37.099 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:52:37.099 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:52:37.099 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:52:37.099 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:52:37.099 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:52:37.099 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:52:37.099 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:52:37.099 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:52:37.099 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:52:37.099 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:52:37.099 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:52:37.100 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:52:37.100 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:52:37.100 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:52:37.100 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:52:37.100 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:52:37.100 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:52:37.100 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:52:37.100 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:52:37.100 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:52:37.100 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:52:37.100 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: main() 2024-01-13T11:52:37.100 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:52:37.101 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:52:37.101 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:52:37.101 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:52:37.101 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:52:37.101 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:52:37.101 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:52:37.101 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:52:37.101 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:52:37.101 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:52:37.101 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:52:37.101 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:52:37.101 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:52:37.101 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:52:37.102 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpan_vrkjj:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp4rhux0v3:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:52:37.102 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: cluster 2024-01-13T11:52:35.313402+0000 mgr.smithi088.xwlxdh (mgr.14182) 185 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:37.102 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: audit 2024-01-13T11:52:36.321031+0000 mon.smithi088 (mon.0) 411 : audit [INF] from='client.? 172.21.15.88:0/3832771019' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7acd71e2-3d05-44e8-aec2-82c4b5b08141"}]: dispatch 2024-01-13T11:52:37.102 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: audit 2024-01-13T11:52:36.328194+0000 mon.smithi088 (mon.0) 412 : audit [INF] from='client.? 172.21.15.88:0/3832771019' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7acd71e2-3d05-44e8-aec2-82c4b5b08141"}]': finished 2024-01-13T11:52:37.102 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: cluster 2024-01-13T11:52:36.328314+0000 mon.smithi088 (mon.0) 413 : cluster [DBG] osdmap e24: 1 total, 0 up, 1 in 2024-01-13T11:52:37.102 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: audit 2024-01-13T11:52:36.328512+0000 mon.smithi088 (mon.0) 414 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:52:37.102 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: audit 2024-01-13T11:52:36.675358+0000 mon.smithi088 (mon.0) 415 : audit [INF] from='client.? 172.21.15.88:0/1815296179' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:52:37.102 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: audit 2024-01-13T11:52:36.681483+0000 mon.smithi088 (mon.0) 416 : audit [INF] from='client.? 172.21.15.88:0/1815296179' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:52:37.102 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:36 smithi088 bash[14069]: cluster 2024-01-13T11:52:36.681525+0000 mon.smithi088 (mon.0) 417 : cluster [DBG] osdmap e25: 0 total, 0 up, 0 in 2024-01-13T11:52:37.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: cephadm 2024-01-13T11:52:35.260736+0000 mgr.smithi088.xwlxdh (mgr.14182) 184 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:52:37.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:52:37.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:52:37.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: return f(*arg) 2024-01-13T11:52:37.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:52:37.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:52:37.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:52:37.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: code, '\n'.join(err))) 2024-01-13T11:52:37.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:52:37.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpan_vrkjj:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp4rhux0v3:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:52:37.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:52:37.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:52:37.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:52:37.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new e639c875-7aba-4495-9772-51448672a2bc 2024-01-13T11:52:37.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:52:37.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:52:37.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:52:37.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:52:37.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:52:37.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:52:37.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:52:37.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:52:37.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:52:37.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:52:37.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:52:37.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:52:37.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:52:37.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:52:37.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:52:37.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:52:37.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:52:37.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:52:37.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:52:37.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:52:37.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:52:37.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:52:37.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:52:37.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:52:37.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:52:37.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:52:37.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:52:37.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:52:37.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:52:37.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:52:37.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:52:37.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:52:37.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:52:37.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:52:37.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:52:37.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:52:37.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:52:37.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:52:37.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:52:37.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:52:37.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:52:37.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:52:37.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:52:37.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:52:37.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:52:37.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:52:37.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:52:37.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:52:37.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:52:37.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:52:37.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:52:37.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:52:37.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:52:37.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:52:37.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:52:37.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:52:37.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:52:37.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:52:37.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: main() 2024-01-13T11:52:37.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:52:37.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:52:37.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:52:37.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:52:37.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:52:37.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:52:37.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:52:37.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:52:37.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:52:37.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:52:37.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:52:37.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:52:37.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:52:37.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:52:37.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpan_vrkjj:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp4rhux0v3:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:52:37.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: cluster 2024-01-13T11:52:35.313402+0000 mgr.smithi088.xwlxdh (mgr.14182) 185 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:37.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: audit 2024-01-13T11:52:36.321031+0000 mon.smithi088 (mon.0) 411 : audit [INF] from='client.? 172.21.15.88:0/3832771019' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7acd71e2-3d05-44e8-aec2-82c4b5b08141"}]: dispatch 2024-01-13T11:52:37.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: audit 2024-01-13T11:52:36.328194+0000 mon.smithi088 (mon.0) 412 : audit [INF] from='client.? 172.21.15.88:0/3832771019' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7acd71e2-3d05-44e8-aec2-82c4b5b08141"}]': finished 2024-01-13T11:52:37.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: cluster 2024-01-13T11:52:36.328314+0000 mon.smithi088 (mon.0) 413 : cluster [DBG] osdmap e24: 1 total, 0 up, 1 in 2024-01-13T11:52:37.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: audit 2024-01-13T11:52:36.328512+0000 mon.smithi088 (mon.0) 414 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:52:37.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: audit 2024-01-13T11:52:36.675358+0000 mon.smithi088 (mon.0) 415 : audit [INF] from='client.? 172.21.15.88:0/1815296179' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:52:37.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: audit 2024-01-13T11:52:36.681483+0000 mon.smithi088 (mon.0) 416 : audit [INF] from='client.? 172.21.15.88:0/1815296179' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:52:37.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:36 smithi138 bash[17620]: cluster 2024-01-13T11:52:36.681525+0000 mon.smithi088 (mon.0) 417 : cluster [DBG] osdmap e25: 0 total, 0 up, 0 in 2024-01-13T11:52:39.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:38 smithi088 bash[14069]: cluster 2024-01-13T11:52:37.313813+0000 mgr.smithi088.xwlxdh (mgr.14182) 186 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:39.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:38 smithi138 bash[17620]: cluster 2024-01-13T11:52:37.313813+0000 mgr.smithi088.xwlxdh (mgr.14182) 186 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:40.286 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: cephadm 2024-01-13T11:52:38.133032+0000 mgr.smithi088.xwlxdh (mgr.14182) 187 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:52:40.286 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:52:40.286 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:52:40.286 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: return f(*arg) 2024-01-13T11:52:40.286 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:52:40.286 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:52:40.286 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:52:40.287 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: code, '\n'.join(err))) 2024-01-13T11:52:40.287 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi088/config 2024-01-13T11:52:40.287 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp9ls8_ujb:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpcxiypytr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:52:40.287 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:52:40.287 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:52:40.287 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:52:40.287 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 7acd71e2-3d05-44e8-aec2-82c4b5b08141 2024-01-13T11:52:40.287 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:52:40.287 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:52:40.287 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:52:40.287 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:52:40.287 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:52:40.287 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:52:40.288 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:52:40.289 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:52:40.289 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:52:40.289 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:52:40.289 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:52:40.289 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:52:40.289 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:52:40.289 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:52:40.289 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:52:40.289 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:52:40.289 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:52:40.289 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:52:40.289 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:52:40.290 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:52:40.290 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:52:40.290 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:52:40.290 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:52:40.290 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:52:40.290 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:52:40.290 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:52:40.290 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:52:40.290 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:52:40.290 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:52:40.290 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:52:40.290 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:52:40.290 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:52:40.291 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:52:40.291 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:52:40.291 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:52:40.291 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:52:40.291 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:52:40.291 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:52:40.291 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:52:40.291 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:52:40.291 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:52:40.291 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:52:40.291 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:52:40.291 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:52:40.291 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:52:40.292 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:52:40.292 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:52:40.292 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:52:40.292 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:52:40.292 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:52:40.292 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:52:40.292 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:52:40.292 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:52:40.292 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:52:40.292 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:52:40.292 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:52:40.292 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:52:40.292 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:52:40.293 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: main() 2024-01-13T11:52:40.293 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:52:40.293 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:52:40.293 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:52:40.293 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:52:40.293 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:52:40.293 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:52:40.293 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:52:40.293 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:52:40.293 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:52:40.293 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:52:40.293 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:52:40.293 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:52:40.294 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:52:40.294 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:52:40.294 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp9ls8_ujb:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpcxiypytr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:52:40.294 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: cephadm 2024-01-13T11:52:38.136560+0000 mgr.smithi088.xwlxdh (mgr.14182) 188 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T11:52:40.294 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: service_id: all-available-devices 2024-01-13T11:52:40.294 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: service_name: osd.all-available-devices 2024-01-13T11:52:40.294 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: placement: 2024-01-13T11:52:40.294 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: host_pattern: '*' 2024-01-13T11:52:40.294 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: spec: 2024-01-13T11:52:40.294 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: data_devices: 2024-01-13T11:52:40.294 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: all: true 2024-01-13T11:52:40.294 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: filter_logic: AND 2024-01-13T11:52:40.294 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: objectstore: bluestore 2024-01-13T11:52:40.295 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:52:40.295 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpan_vrkjj:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp4rhux0v3:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:52:40.295 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:52:40.295 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:52:40.295 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:52:40.295 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new e639c875-7aba-4495-9772-51448672a2bc 2024-01-13T11:52:40.295 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:52:40.295 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:52:40.295 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:52:40.295 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:52:40.295 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:52:40.295 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:52:40.295 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:52:40.296 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:52:40.296 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:52:40.296 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:52:40.296 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:52:40.296 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:52:40.296 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:52:40.296 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:52:40.296 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:52:40.296 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:52:40.296 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:52:40.296 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:52:40.296 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:52:40.296 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:52:40.297 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:52:40.297 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:52:40.297 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:52:40.297 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:52:40.297 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:52:40.297 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:52:40.297 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:52:40.297 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:52:40.297 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:52:40.297 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:52:40.297 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:52:40.297 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:52:40.297 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:52:40.297 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:52:40.298 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:52:40.298 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:52:40.298 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:52:40.298 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:52:40.298 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:52:40.298 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:52:40.298 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:52:40.298 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:52:40.298 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:52:40.298 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:52:40.298 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:52:40.298 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:52:40.298 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:52:40.298 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:52:40.299 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:52:40.299 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:52:40.299 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:52:40.299 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:52:40.299 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:52:40.299 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:52:40.299 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:52:40.299 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:52:40.299 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:52:40.299 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:52:40.299 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: main() 2024-01-13T11:52:40.300 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:52:40.300 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:52:40.300 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:52:40.300 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:52:40.300 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:52:40.300 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:52:40.300 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:52:40.300 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:52:40.300 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:52:40.300 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:52:40.300 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:52:40.300 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:52:40.301 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:52:40.301 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:52:40.301 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpan_vrkjj:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp4rhux0v3:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:52:40.301 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:52:40.301 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T11:52:40.301 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: if self._apply_service(spec): 2024-01-13T11:52:40.301 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T11:52:40.301 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T11:52:40.301 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T11:52:40.301 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T11:52:40.301 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T11:52:40.301 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T11:52:40.301 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T11:52:40.301 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T11:52:40.302 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T11:52:40.302 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: raise self._value 2024-01-13T11:52:40.302 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T11:52:40.302 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: result = (True, func(*args, **kwds)) 2024-01-13T11:52:40.302 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T11:52:40.302 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: return list(map(*args)) 2024-01-13T11:52:40.302 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:52:40.302 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: return f(*arg) 2024-01-13T11:52:40.302 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:52:40.302 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:52:40.303 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:52:40.303 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: code, '\n'.join(err))) 2024-01-13T11:52:40.303 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:52:40.303 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpan_vrkjj:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp4rhux0v3:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:52:40.303 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:52:40.303 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:52:40.303 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:52:40.304 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new e639c875-7aba-4495-9772-51448672a2bc 2024-01-13T11:52:40.304 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:52:40.304 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:52:40.304 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:52:40.304 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:52:40.304 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:52:40.304 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:52:40.304 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:52:40.304 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:52:40.304 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:52:40.304 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:52:40.304 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:52:40.305 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:52:40.305 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:52:40.305 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:52:40.305 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:52:40.305 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:52:40.305 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:52:40.305 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:52:40.305 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:52:40.305 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:52:40.305 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:52:40.305 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:52:40.305 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:52:40.305 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:52:40.305 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:52:40.306 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:52:40.306 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:52:40.306 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:52:40.306 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:52:40.306 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:52:40.306 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:52:40.306 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:52:40.306 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:52:40.306 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:52:40.306 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:52:40.306 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:52:40.306 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:52:40.306 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:52:40.306 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:52:40.307 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:52:40.307 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:52:40.307 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:52:40.307 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:52:40.307 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:52:40.307 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:52:40.307 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:52:40.307 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:52:40.307 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:52:40.307 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:52:40.307 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:52:40.307 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:52:40.307 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:52:40.307 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:52:40.308 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:52:40.308 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:52:40.308 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:52:40.308 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:52:40.308 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:52:40.308 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: main() 2024-01-13T11:52:40.308 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:52:40.308 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:52:40.308 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:52:40.308 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:52:40.308 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:52:40.308 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:52:40.308 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:52:40.308 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:52:40.309 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:52:40.309 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:52:40.309 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:52:40.309 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:52:40.309 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:52:40.309 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:52:40.309 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpan_vrkjj:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp4rhux0v3:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:52:40.309 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: cluster 2024-01-13T11:52:38.137571+0000 mgr.smithi088.xwlxdh (mgr.14182) 189 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:40.309 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:39 smithi088 bash[14069]: cluster 2024-01-13T11:52:38.937842+0000 mon.smithi088 (mon.0) 418 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T11:52:40.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: cephadm 2024-01-13T11:52:38.133032+0000 mgr.smithi088.xwlxdh (mgr.14182) 187 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:52:40.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:52:40.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:52:40.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: return f(*arg) 2024-01-13T11:52:40.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:52:40.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:52:40.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:52:40.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: code, '\n'.join(err))) 2024-01-13T11:52:40.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi088/config 2024-01-13T11:52:40.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp9ls8_ujb:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpcxiypytr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:52:40.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:52:40.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:52:40.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:52:40.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 7acd71e2-3d05-44e8-aec2-82c4b5b08141 2024-01-13T11:52:40.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:52:40.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:52:40.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:52:40.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:52:40.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:52:40.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:52:40.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:52:40.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:52:40.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:52:40.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:52:40.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:52:40.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:52:40.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:52:40.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:52:40.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:52:40.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:52:40.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:52:40.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:52:40.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:52:40.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:52:40.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:52:40.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:52:40.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:52:40.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:52:40.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:52:40.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:52:40.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:52:40.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:52:40.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:52:40.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:52:40.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:52:40.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:52:40.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:52:40.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:52:40.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:52:40.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:52:40.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:52:40.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:52:40.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:52:40.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:52:40.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:52:40.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:52:40.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:52:40.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:52:40.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:52:40.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:52:40.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:52:40.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:52:40.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:52:40.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:52:40.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:52:40.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:52:40.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:52:40.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:52:40.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:52:40.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:52:40.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:52:40.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:52:40.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: main() 2024-01-13T11:52:40.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:52:40.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:52:40.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:52:40.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:52:40.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:52:40.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:52:40.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:52:40.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:52:40.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:52:40.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:52:40.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:52:40.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:52:40.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:52:40.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:52:40.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp9ls8_ujb:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpcxiypytr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:52:40.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: cephadm 2024-01-13T11:52:38.136560+0000 mgr.smithi088.xwlxdh (mgr.14182) 188 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T11:52:40.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: service_id: all-available-devices 2024-01-13T11:52:40.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: service_name: osd.all-available-devices 2024-01-13T11:52:40.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: placement: 2024-01-13T11:52:40.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: host_pattern: '*' 2024-01-13T11:52:40.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: spec: 2024-01-13T11:52:40.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: data_devices: 2024-01-13T11:52:40.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: all: true 2024-01-13T11:52:40.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: filter_logic: AND 2024-01-13T11:52:40.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: objectstore: bluestore 2024-01-13T11:52:40.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:52:40.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpan_vrkjj:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp4rhux0v3:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:52:40.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:52:40.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:52:40.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:52:40.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new e639c875-7aba-4495-9772-51448672a2bc 2024-01-13T11:52:40.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:52:40.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:52:40.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:52:40.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:52:40.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:52:40.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:52:40.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:52:40.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:52:40.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:52:40.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:52:40.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:52:40.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:52:40.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:52:40.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:52:40.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:52:40.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:52:40.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:52:40.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:52:40.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:52:40.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:52:40.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:52:40.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:52:40.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:52:40.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:52:40.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:52:40.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:52:40.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:52:40.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:52:40.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:52:40.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:52:40.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:52:40.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:52:40.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:52:40.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:52:40.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:52:40.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:52:40.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:52:40.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:52:40.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:52:40.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:52:40.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:52:40.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:52:40.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:52:40.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:52:40.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:52:40.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:52:40.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:52:40.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:52:40.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:52:40.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:52:40.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:52:40.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:52:40.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:52:40.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:52:40.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:52:40.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:52:40.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:52:40.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:52:40.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: main() 2024-01-13T11:52:40.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:52:40.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:52:40.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:52:40.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:52:40.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:52:40.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:52:40.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:52:40.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:52:40.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:52:40.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:52:40.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:52:40.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:52:40.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:52:40.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:52:40.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpan_vrkjj:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp4rhux0v3:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:52:40.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:52:40.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T11:52:40.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: if self._apply_service(spec): 2024-01-13T11:52:40.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T11:52:40.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T11:52:40.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T11:52:40.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T11:52:40.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T11:52:40.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T11:52:40.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T11:52:40.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T11:52:40.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T11:52:40.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: raise self._value 2024-01-13T11:52:40.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T11:52:40.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: result = (True, func(*args, **kwds)) 2024-01-13T11:52:40.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T11:52:40.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: return list(map(*args)) 2024-01-13T11:52:40.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:52:40.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: return f(*arg) 2024-01-13T11:52:40.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:52:40.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:52:40.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:52:40.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: code, '\n'.join(err))) 2024-01-13T11:52:40.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:52:40.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpan_vrkjj:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp4rhux0v3:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:52:40.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:52:40.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:52:40.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:52:40.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new e639c875-7aba-4495-9772-51448672a2bc 2024-01-13T11:52:40.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:52:40.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:52:40.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:52:40.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:52:40.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:52:40.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:52:40.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:52:40.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:52:40.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:52:40.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:52:40.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:52:40.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:52:40.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:52:40.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:52:40.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:52:40.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:52:40.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:52:40.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:52:40.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:52:40.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:52:40.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:52:40.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:52:40.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:52:40.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:52:40.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:52:40.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:52:40.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:52:40.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:52:40.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:52:40.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:52:40.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:52:40.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:52:40.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:52:40.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:52:40.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:52:40.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:52:40.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:52:40.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:52:40.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:52:40.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:52:40.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:52:40.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:52:40.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:52:40.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:52:40.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:52:40.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:52:40.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:52:40.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:52:40.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:52:40.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:52:40.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:52:40.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:52:40.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:52:40.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:52:40.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:52:40.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:52:40.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:52:40.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:52:40.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: main() 2024-01-13T11:52:40.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:52:40.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:52:40.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:52:40.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:52:40.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:52:40.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:52:40.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:52:40.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:52:40.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:52:40.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:52:40.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:52:40.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:52:40.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:52:40.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:52:40.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpan_vrkjj:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp4rhux0v3:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:52:40.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: cluster 2024-01-13T11:52:38.137571+0000 mgr.smithi088.xwlxdh (mgr.14182) 189 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:40.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:39 smithi138 bash[17620]: cluster 2024-01-13T11:52:38.937842+0000 mon.smithi088 (mon.0) 418 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T11:52:40.688 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:52:40.992 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:40 smithi088 bash[14069]: audit 2024-01-13T11:52:40.698789+0000 mon.smithi088 (mon.0) 419 : audit [DBG] from='client.? 172.21.15.88:0/3365241932' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:52:41.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:40 smithi138 bash[17620]: audit 2024-01-13T11:52:40.698789+0000 mon.smithi088 (mon.0) 419 : audit [DBG] from='client.? 172.21.15.88:0/3365241932' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:52:41.537 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:52:42.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:41 smithi138 bash[17620]: cluster 2024-01-13T11:52:40.138459+0000 mgr.smithi088.xwlxdh (mgr.14182) 190 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:42.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:41 smithi088 bash[14069]: cluster 2024-01-13T11:52:40.138459+0000 mgr.smithi088.xwlxdh (mgr.14182) 190 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:42.538 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:52:44.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:43 smithi138 bash[17620]: cluster 2024-01-13T11:52:42.139506+0000 mgr.smithi088.xwlxdh (mgr.14182) 191 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:44.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:43 smithi088 bash[14069]: cluster 2024-01-13T11:52:42.139506+0000 mgr.smithi088.xwlxdh (mgr.14182) 191 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:45.764 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:52:45.982 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:45 smithi088 bash[14069]: cluster 2024-01-13T11:52:44.140533+0000 mgr.smithi088.xwlxdh (mgr.14182) 192 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:45.982 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:45 smithi088 bash[14069]: audit 2024-01-13T11:52:44.732457+0000 mon.smithi088 (mon.0) 420 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:52:46.123 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:45 smithi138 bash[17620]: cluster 2024-01-13T11:52:44.140533+0000 mgr.smithi088.xwlxdh (mgr.14182) 192 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:46.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:45 smithi138 bash[17620]: audit 2024-01-13T11:52:44.732457+0000 mon.smithi088 (mon.0) 420 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:52:46.504 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:52:46.891 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:46 smithi088 bash[14069]: audit 2024-01-13T11:52:45.775672+0000 mon.smithi088 (mon.0) 421 : audit [DBG] from='client.? 172.21.15.88:0/820328062' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:52:47.123 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:46 smithi138 bash[17620]: audit 2024-01-13T11:52:45.775672+0000 mon.smithi088 (mon.0) 421 : audit [DBG] from='client.? 172.21.15.88:0/820328062' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:52:47.505 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:52:47.891 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:47 smithi088 bash[14069]: cluster 2024-01-13T11:52:46.141546+0000 mgr.smithi088.xwlxdh (mgr.14182) 193 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:48.123 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:47 smithi138 bash[17620]: cluster 2024-01-13T11:52:46.141546+0000 mgr.smithi088.xwlxdh (mgr.14182) 193 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:50.242 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:49 smithi088 bash[14069]: cluster 2024-01-13T11:52:48.142595+0000 mgr.smithi088.xwlxdh (mgr.14182) 194 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:50.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:49 smithi138 bash[17620]: cluster 2024-01-13T11:52:48.142595+0000 mgr.smithi088.xwlxdh (mgr.14182) 194 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:50.588 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:52:51.121 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:50 smithi088 bash[14069]: audit 2024-01-13T11:52:50.598936+0000 mon.smithi088 (mon.0) 422 : audit [DBG] from='client.? 172.21.15.88:0/3642838275' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:52:51.301 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:52:51.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:50 smithi138 bash[17620]: audit 2024-01-13T11:52:50.598936+0000 mon.smithi088 (mon.0) 422 : audit [DBG] from='client.? 172.21.15.88:0/3642838275' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:52:52.303 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:52:52.315 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:51 smithi088 bash[14069]: cluster 2024-01-13T11:52:50.143598+0000 mgr.smithi088.xwlxdh (mgr.14182) 195 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:52.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:51 smithi138 bash[17620]: cluster 2024-01-13T11:52:50.143598+0000 mgr.smithi088.xwlxdh (mgr.14182) 195 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:54.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:53 smithi088 bash[14069]: cluster 2024-01-13T11:52:52.144634+0000 mgr.smithi088.xwlxdh (mgr.14182) 196 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:54.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:53 smithi138 bash[17620]: cluster 2024-01-13T11:52:52.144634+0000 mgr.smithi088.xwlxdh (mgr.14182) 196 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:55.437 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:52:56.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:55 smithi088 bash[14069]: cluster 2024-01-13T11:52:54.145571+0000 mgr.smithi088.xwlxdh (mgr.14182) 197 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:56.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:55 smithi088 bash[14069]: audit 2024-01-13T11:52:55.448600+0000 mon.smithi088 (mon.0) 423 : audit [DBG] from='client.? 172.21.15.88:0/588746766' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:52:56.219 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:52:56.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:55 smithi138 bash[17620]: cluster 2024-01-13T11:52:54.145571+0000 mgr.smithi088.xwlxdh (mgr.14182) 197 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:56.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:55 smithi138 bash[17620]: audit 2024-01-13T11:52:55.448600+0000 mon.smithi088 (mon.0) 423 : audit [DBG] from='client.? 172.21.15.88:0/588746766' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:52:57.220 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:52:58.279 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:57 smithi088 bash[14069]: cluster 2024-01-13T11:52:56.146616+0000 mgr.smithi088.xwlxdh (mgr.14182) 198 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:52:58.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:57 smithi138 bash[17620]: cluster 2024-01-13T11:52:56.146616+0000 mgr.smithi088.xwlxdh (mgr.14182) 198 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:00.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:52:59 smithi138 bash[17620]: cluster 2024-01-13T11:52:58.147684+0000 mgr.smithi088.xwlxdh (mgr.14182) 199 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:00.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:52:59 smithi088 bash[14069]: cluster 2024-01-13T11:52:58.147684+0000 mgr.smithi088.xwlxdh (mgr.14182) 199 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:00.397 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:53:01.137 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:53:01.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:00 smithi138 bash[17620]: audit 2024-01-13T11:53:00.408365+0000 mon.smithi088 (mon.0) 424 : audit [DBG] from='client.? 172.21.15.88:0/3602676132' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:53:01.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:00 smithi088 bash[14069]: audit 2024-01-13T11:53:00.408365+0000 mon.smithi088 (mon.0) 424 : audit [DBG] from='client.? 172.21.15.88:0/3602676132' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:53:02.139 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:53:02.335 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:01 smithi088 bash[14069]: cluster 2024-01-13T11:53:00.148318+0000 mgr.smithi088.xwlxdh (mgr.14182) 200 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:02.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:01 smithi138 bash[17620]: cluster 2024-01-13T11:53:00.148318+0000 mgr.smithi088.xwlxdh (mgr.14182) 200 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:04.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:03 smithi138 bash[17620]: cluster 2024-01-13T11:53:02.149312+0000 mgr.smithi088.xwlxdh (mgr.14182) 201 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:04.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:03 smithi088 bash[14069]: cluster 2024-01-13T11:53:02.149312+0000 mgr.smithi088.xwlxdh (mgr.14182) 201 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:05.295 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:53:05.328 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:05 smithi088 bash[14069]: cluster 2024-01-13T11:53:04.150036+0000 mgr.smithi088.xwlxdh (mgr.14182) 202 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:05.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:05 smithi138 bash[17620]: cluster 2024-01-13T11:53:04.150036+0000 mgr.smithi088.xwlxdh (mgr.14182) 202 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:06.099 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:53:06.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:06 smithi138 bash[17620]: audit 2024-01-13T11:53:05.306705+0000 mon.smithi088 (mon.0) 425 : audit [DBG] from='client.? 172.21.15.88:0/3209195570' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:53:06.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:06 smithi088 bash[14069]: audit 2024-01-13T11:53:05.306705+0000 mon.smithi088 (mon.0) 425 : audit [DBG] from='client.? 172.21.15.88:0/3209195570' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:53:07.100 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:53:07.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:07 smithi138 bash[17620]: cluster 2024-01-13T11:53:06.151092+0000 mgr.smithi088.xwlxdh (mgr.14182) 203 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:07.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:07 smithi088 bash[14069]: cluster 2024-01-13T11:53:06.151092+0000 mgr.smithi088.xwlxdh (mgr.14182) 203 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:10.001 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:53:10.250 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:09 smithi088 bash[14069]: cluster 2024-01-13T11:53:08.152150+0000 mgr.smithi088.xwlxdh (mgr.14182) 204 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:10.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:09 smithi138 bash[17620]: cluster 2024-01-13T11:53:08.152150+0000 mgr.smithi088.xwlxdh (mgr.14182) 204 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:10.720 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:53:11.121 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:10 smithi088 bash[14069]: audit 2024-01-13T11:53:10.011937+0000 mon.smithi088 (mon.0) 426 : audit [DBG] from='client.? 172.21.15.88:0/48338208' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:53:11.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:10 smithi138 bash[17620]: audit 2024-01-13T11:53:10.011937+0000 mon.smithi088 (mon.0) 426 : audit [DBG] from='client.? 172.21.15.88:0/48338208' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:53:11.776 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:53:12.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:11 smithi088 bash[14069]: cluster 2024-01-13T11:53:10.153185+0000 mgr.smithi088.xwlxdh (mgr.14182) 205 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:12.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:11 smithi138 bash[17620]: cluster 2024-01-13T11:53:10.153185+0000 mgr.smithi088.xwlxdh (mgr.14182) 205 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:14.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:13 smithi088 bash[14069]: cluster 2024-01-13T11:53:12.153849+0000 mgr.smithi088.xwlxdh (mgr.14182) 206 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:14.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:13 smithi138 bash[17620]: cluster 2024-01-13T11:53:12.153849+0000 mgr.smithi088.xwlxdh (mgr.14182) 206 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:14.897 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:53:15.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:14 smithi138 bash[17620]: audit 2024-01-13T11:53:14.907557+0000 mon.smithi088 (mon.0) 427 : audit [DBG] from='client.? 172.21.15.88:0/3219702728' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:53:15.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:14 smithi088 bash[14069]: audit 2024-01-13T11:53:14.907557+0000 mon.smithi088 (mon.0) 427 : audit [DBG] from='client.? 172.21.15.88:0/3219702728' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:53:15.658 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:53:16.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:15 smithi138 bash[17620]: cluster 2024-01-13T11:53:14.154465+0000 mgr.smithi088.xwlxdh (mgr.14182) 207 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:16.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:15 smithi088 bash[14069]: cluster 2024-01-13T11:53:14.154465+0000 mgr.smithi088.xwlxdh (mgr.14182) 207 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:16.659 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:53:18.040 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:17 smithi088 bash[14069]: cluster 2024-01-13T11:53:16.155423+0000 mgr.smithi088.xwlxdh (mgr.14182) 208 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:18.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:17 smithi138 bash[17620]: cluster 2024-01-13T11:53:16.155423+0000 mgr.smithi088.xwlxdh (mgr.14182) 208 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:19.806 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:53:20.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:19 smithi088 bash[14069]: cluster 2024-01-13T11:53:18.156468+0000 mgr.smithi088.xwlxdh (mgr.14182) 209 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:20.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:19 smithi088 bash[14069]: audit 2024-01-13T11:53:19.817589+0000 mon.smithi088 (mon.0) 428 : audit [DBG] from='client.? 172.21.15.88:0/587668519' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:53:20.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:19 smithi138 bash[17620]: cluster 2024-01-13T11:53:18.156468+0000 mgr.smithi088.xwlxdh (mgr.14182) 209 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:20.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:19 smithi138 bash[17620]: audit 2024-01-13T11:53:19.817589+0000 mon.smithi088 (mon.0) 428 : audit [DBG] from='client.? 172.21.15.88:0/587668519' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:53:20.577 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:53:21.578 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:53:22.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:21 smithi088 bash[14069]: cluster 2024-01-13T11:53:20.157503+0000 mgr.smithi088.xwlxdh (mgr.14182) 210 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:22.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:21 smithi138 bash[17620]: cluster 2024-01-13T11:53:20.157503+0000 mgr.smithi088.xwlxdh (mgr.14182) 210 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:24.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:23 smithi088 bash[14069]: cluster 2024-01-13T11:53:22.158428+0000 mgr.smithi088.xwlxdh (mgr.14182) 211 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:24.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:23 smithi138 bash[17620]: cluster 2024-01-13T11:53:22.158428+0000 mgr.smithi088.xwlxdh (mgr.14182) 211 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:24.941 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:53:25.231 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:24 smithi088 bash[14069]: audit 2024-01-13T11:53:24.952841+0000 mon.smithi088 (mon.0) 429 : audit [DBG] from='client.? 172.21.15.88:0/1240698674' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:53:25.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:24 smithi138 bash[17620]: audit 2024-01-13T11:53:24.952841+0000 mon.smithi088 (mon.0) 429 : audit [DBG] from='client.? 172.21.15.88:0/1240698674' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:53:25.872 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:53:26.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:25 smithi088 bash[14069]: cluster 2024-01-13T11:53:24.159412+0000 mgr.smithi088.xwlxdh (mgr.14182) 212 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:26.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:25 smithi138 bash[17620]: cluster 2024-01-13T11:53:24.159412+0000 mgr.smithi088.xwlxdh (mgr.14182) 212 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:26.873 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:53:27.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:26 smithi088 bash[14069]: cluster 2024-01-13T11:53:26.160417+0000 mgr.smithi088.xwlxdh (mgr.14182) 213 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:27.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:26 smithi138 bash[17620]: cluster 2024-01-13T11:53:26.160417+0000 mgr.smithi088.xwlxdh (mgr.14182) 213 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:30.094 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:53:30.122 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:29 smithi088 bash[14069]: cluster 2024-01-13T11:53:28.161483+0000 mgr.smithi088.xwlxdh (mgr.14182) 214 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:30.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:29 smithi138 bash[17620]: cluster 2024-01-13T11:53:28.161483+0000 mgr.smithi088.xwlxdh (mgr.14182) 214 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:30.855 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:53:31.121 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:30 smithi088 bash[14069]: audit 2024-01-13T11:53:30.105209+0000 mon.smithi088 (mon.0) 430 : audit [DBG] from='client.? 172.21.15.88:0/3235899783' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:53:31.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:30 smithi138 bash[17620]: audit 2024-01-13T11:53:30.105209+0000 mon.smithi088 (mon.0) 430 : audit [DBG] from='client.? 172.21.15.88:0/3235899783' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:53:31.856 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:53:32.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:31 smithi088 bash[14069]: cluster 2024-01-13T11:53:30.162473+0000 mgr.smithi088.xwlxdh (mgr.14182) 215 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:32.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:31 smithi138 bash[17620]: cluster 2024-01-13T11:53:30.162473+0000 mgr.smithi088.xwlxdh (mgr.14182) 215 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:34.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:33 smithi088 bash[14069]: cluster 2024-01-13T11:53:32.163464+0000 mgr.smithi088.xwlxdh (mgr.14182) 216 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:34.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:33 smithi138 bash[17620]: cluster 2024-01-13T11:53:32.163464+0000 mgr.smithi088.xwlxdh (mgr.14182) 216 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:35.049 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:53:35.806 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:53:36.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:35 smithi088 bash[14069]: cluster 2024-01-13T11:53:34.164435+0000 mgr.smithi088.xwlxdh (mgr.14182) 217 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:36.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:35 smithi088 bash[14069]: audit 2024-01-13T11:53:35.060334+0000 mon.smithi088 (mon.0) 431 : audit [DBG] from='client.? 172.21.15.88:0/3271161503' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:53:36.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:35 smithi138 bash[17620]: cluster 2024-01-13T11:53:34.164435+0000 mgr.smithi088.xwlxdh (mgr.14182) 217 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:36.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:35 smithi138 bash[17620]: audit 2024-01-13T11:53:35.060334+0000 mon.smithi088 (mon.0) 431 : audit [DBG] from='client.? 172.21.15.88:0/3271161503' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:53:36.807 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:53:38.344 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:37 smithi088 bash[14069]: cluster 2024-01-13T11:53:36.165422+0000 mgr.smithi088.xwlxdh (mgr.14182) 218 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:38.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:37 smithi138 bash[17620]: cluster 2024-01-13T11:53:36.165422+0000 mgr.smithi088.xwlxdh (mgr.14182) 218 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:39.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:38 smithi138 bash[17620]: audit 2024-01-13T11:53:38.140182+0000 mon.smithi088 (mon.0) 432 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:53:39.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:38 smithi138 bash[17620]: audit 2024-01-13T11:53:38.415749+0000 mon.smithi088 (mon.0) 433 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:53:39.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:38 smithi138 bash[17620]: audit 2024-01-13T11:53:38.416876+0000 mon.smithi088 (mon.0) 434 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T11:53:39.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:38 smithi138 bash[17620]: audit 2024-01-13T11:53:38.424173+0000 mon.smithi088 (mon.0) 435 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:53:39.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:38 smithi138 bash[17620]: audit 2024-01-13T11:53:38.431606+0000 mon.smithi088 (mon.0) 436 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T11:53:39.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:38 smithi138 bash[17620]: audit 2024-01-13T11:53:38.436692+0000 mon.smithi088 (mon.0) 437 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:53:39.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:38 smithi138 bash[17620]: audit 2024-01-13T11:53:38.437952+0000 mon.smithi088 (mon.0) 438 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:53:39.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:38 smithi138 bash[17620]: audit 2024-01-13T11:53:38.438349+0000 mon.smithi088 (mon.0) 439 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:53:39.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:38 smithi138 bash[17620]: audit 2024-01-13T11:53:38.438905+0000 mon.smithi088 (mon.0) 440 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:53:39.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:38 smithi088 bash[14069]: audit 2024-01-13T11:53:38.140182+0000 mon.smithi088 (mon.0) 432 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:53:39.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:38 smithi088 bash[14069]: audit 2024-01-13T11:53:38.415749+0000 mon.smithi088 (mon.0) 433 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:53:39.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:38 smithi088 bash[14069]: audit 2024-01-13T11:53:38.416876+0000 mon.smithi088 (mon.0) 434 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T11:53:39.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:38 smithi088 bash[14069]: audit 2024-01-13T11:53:38.424173+0000 mon.smithi088 (mon.0) 435 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:53:39.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:38 smithi088 bash[14069]: audit 2024-01-13T11:53:38.431606+0000 mon.smithi088 (mon.0) 436 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T11:53:39.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:38 smithi088 bash[14069]: audit 2024-01-13T11:53:38.436692+0000 mon.smithi088 (mon.0) 437 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:53:39.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:38 smithi088 bash[14069]: audit 2024-01-13T11:53:38.437952+0000 mon.smithi088 (mon.0) 438 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:53:39.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:38 smithi088 bash[14069]: audit 2024-01-13T11:53:38.438349+0000 mon.smithi088 (mon.0) 439 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:53:39.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:38 smithi088 bash[14069]: audit 2024-01-13T11:53:38.438905+0000 mon.smithi088 (mon.0) 440 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:53:40.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:39 smithi088 bash[14069]: cluster 2024-01-13T11:53:38.166211+0000 mgr.smithi088.xwlxdh (mgr.14182) 219 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:40.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:39 smithi088 bash[14069]: cluster 2024-01-13T11:53:38.424761+0000 mgr.smithi088.xwlxdh (mgr.14182) 220 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:40.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:39 smithi088 bash[14069]: cluster 2024-01-13T11:53:38.960561+0000 mon.smithi088 (mon.0) 441 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T11:53:40.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:39 smithi088 bash[14069]: cluster 2024-01-13T11:53:38.960597+0000 mon.smithi088 (mon.0) 442 : cluster [INF] Cluster is now healthy 2024-01-13T11:53:40.169 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:53:40.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:39 smithi138 bash[17620]: cluster 2024-01-13T11:53:38.166211+0000 mgr.smithi088.xwlxdh (mgr.14182) 219 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:40.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:39 smithi138 bash[17620]: cluster 2024-01-13T11:53:38.424761+0000 mgr.smithi088.xwlxdh (mgr.14182) 220 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:40.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:39 smithi138 bash[17620]: cluster 2024-01-13T11:53:38.960561+0000 mon.smithi088 (mon.0) 441 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T11:53:40.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:39 smithi138 bash[17620]: cluster 2024-01-13T11:53:38.960597+0000 mon.smithi088 (mon.0) 442 : cluster [INF] Cluster is now healthy 2024-01-13T11:53:40.940 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:53:41.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:40 smithi138 bash[17620]: audit 2024-01-13T11:53:40.180774+0000 mon.smithi088 (mon.0) 443 : audit [DBG] from='client.? 172.21.15.88:0/1828386463' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:53:41.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:40 smithi088 bash[14069]: audit 2024-01-13T11:53:40.180774+0000 mon.smithi088 (mon.0) 443 : audit [DBG] from='client.? 172.21.15.88:0/1828386463' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:53:41.942 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:53:42.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:41 smithi138 bash[17620]: cluster 2024-01-13T11:53:40.425102+0000 mgr.smithi088.xwlxdh (mgr.14182) 221 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:42.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:41 smithi088 bash[14069]: cluster 2024-01-13T11:53:40.425102+0000 mgr.smithi088.xwlxdh (mgr.14182) 221 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:44.123 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:43 smithi138 bash[17620]: cluster 2024-01-13T11:53:42.425533+0000 mgr.smithi088.xwlxdh (mgr.14182) 222 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:44.215 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:43 smithi088 bash[14069]: cluster 2024-01-13T11:53:42.425533+0000 mgr.smithi088.xwlxdh (mgr.14182) 222 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:45.670 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:53:45.873 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:45 smithi138 bash[17620]: cluster 2024-01-13T11:53:44.426007+0000 mgr.smithi088.xwlxdh (mgr.14182) 223 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:45.873 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:45 smithi138 bash[17620]: audit 2024-01-13T11:53:44.733555+0000 mon.smithi088 (mon.0) 444 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:53:45.873 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:45 smithi138 bash[17620]: audit 2024-01-13T11:53:45.681608+0000 mon.smithi088 (mon.0) 445 : audit [DBG] from='client.? 172.21.15.88:0/3544076614' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:53:45.891 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:45 smithi088 bash[14069]: cluster 2024-01-13T11:53:44.426007+0000 mgr.smithi088.xwlxdh (mgr.14182) 223 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:45.891 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:45 smithi088 bash[14069]: audit 2024-01-13T11:53:44.733555+0000 mon.smithi088 (mon.0) 444 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:53:45.891 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:45 smithi088 bash[14069]: audit 2024-01-13T11:53:45.681608+0000 mon.smithi088 (mon.0) 445 : audit [DBG] from='client.? 172.21.15.88:0/3544076614' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:53:47.064 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:53:48.065 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:53:48.078 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:47 smithi088 bash[14069]: cluster 2024-01-13T11:53:46.426491+0000 mgr.smithi088.xwlxdh (mgr.14182) 224 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:48.078 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:47 smithi088 bash[14069]: audit 2024-01-13T11:53:47.243366+0000 mon.smithi138 (mon.1) 11 : audit [INF] from='client.? 172.21.15.138:0/3992810074' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "24553668-7d79-4dfd-bd47-7eebe1d6b96b"}]: dispatch 2024-01-13T11:53:48.078 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:47 smithi088 bash[14069]: audit 2024-01-13T11:53:47.244067+0000 mon.smithi088 (mon.0) 446 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "24553668-7d79-4dfd-bd47-7eebe1d6b96b"}]: dispatch 2024-01-13T11:53:48.078 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:47 smithi088 bash[14069]: audit 2024-01-13T11:53:47.252866+0000 mon.smithi088 (mon.0) 447 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "24553668-7d79-4dfd-bd47-7eebe1d6b96b"}]': finished 2024-01-13T11:53:48.078 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:47 smithi088 bash[14069]: cluster 2024-01-13T11:53:47.252947+0000 mon.smithi088 (mon.0) 448 : cluster [DBG] osdmap e26: 1 total, 0 up, 1 in 2024-01-13T11:53:48.079 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:47 smithi088 bash[14069]: audit 2024-01-13T11:53:47.253125+0000 mon.smithi088 (mon.0) 449 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:53:48.079 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:47 smithi088 bash[14069]: audit 2024-01-13T11:53:47.596882+0000 mon.smithi138 (mon.1) 12 : audit [INF] from='client.? 172.21.15.138:0/1069621294' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:53:48.079 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:47 smithi088 bash[14069]: audit 2024-01-13T11:53:47.597511+0000 mon.smithi088 (mon.0) 450 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:53:48.079 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:47 smithi088 bash[14069]: audit 2024-01-13T11:53:47.607753+0000 mon.smithi088 (mon.0) 451 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:53:48.079 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:47 smithi088 bash[14069]: cluster 2024-01-13T11:53:47.607842+0000 mon.smithi088 (mon.0) 452 : cluster [DBG] osdmap e27: 0 total, 0 up, 0 in 2024-01-13T11:53:48.123 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:47 smithi138 bash[17620]: cluster 2024-01-13T11:53:46.426491+0000 mgr.smithi088.xwlxdh (mgr.14182) 224 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:48.123 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:47 smithi138 bash[17620]: audit 2024-01-13T11:53:47.243366+0000 mon.smithi138 (mon.1) 11 : audit [INF] from='client.? 172.21.15.138:0/3992810074' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "24553668-7d79-4dfd-bd47-7eebe1d6b96b"}]: dispatch 2024-01-13T11:53:48.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:47 smithi138 bash[17620]: audit 2024-01-13T11:53:47.244067+0000 mon.smithi088 (mon.0) 446 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "24553668-7d79-4dfd-bd47-7eebe1d6b96b"}]: dispatch 2024-01-13T11:53:48.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:47 smithi138 bash[17620]: audit 2024-01-13T11:53:47.252866+0000 mon.smithi088 (mon.0) 447 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "24553668-7d79-4dfd-bd47-7eebe1d6b96b"}]': finished 2024-01-13T11:53:48.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:47 smithi138 bash[17620]: cluster 2024-01-13T11:53:47.252947+0000 mon.smithi088 (mon.0) 448 : cluster [DBG] osdmap e26: 1 total, 0 up, 1 in 2024-01-13T11:53:48.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:47 smithi138 bash[17620]: audit 2024-01-13T11:53:47.253125+0000 mon.smithi088 (mon.0) 449 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:53:48.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:47 smithi138 bash[17620]: audit 2024-01-13T11:53:47.596882+0000 mon.smithi138 (mon.1) 12 : audit [INF] from='client.? 172.21.15.138:0/1069621294' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:53:48.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:47 smithi138 bash[17620]: audit 2024-01-13T11:53:47.597511+0000 mon.smithi088 (mon.0) 450 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:53:48.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:47 smithi138 bash[17620]: audit 2024-01-13T11:53:47.607753+0000 mon.smithi088 (mon.0) 451 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:53:48.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:47 smithi138 bash[17620]: cluster 2024-01-13T11:53:47.607842+0000 mon.smithi088 (mon.0) 452 : cluster [DBG] osdmap e27: 0 total, 0 up, 0 in 2024-01-13T11:53:50.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: cluster 2024-01-13T11:53:48.426875+0000 mgr.smithi088.xwlxdh (mgr.14182) 225 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:50.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: cephadm 2024-01-13T11:53:48.452917+0000 mgr.smithi088.xwlxdh (mgr.14182) 226 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:53:50.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:53:50.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:53:50.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: return f(*arg) 2024-01-13T11:53:50.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:53:50.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:53:50.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:53:50.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: code, '\n'.join(err))) 2024-01-13T11:53:50.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:53:50.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpopuohfub:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpx0l9j9xt:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:53:50.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:53:50.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:53:50.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:53:50.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 24553668-7d79-4dfd-bd47-7eebe1d6b96b 2024-01-13T11:53:50.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:53:50.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:53:50.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:53:50.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:53:50.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:53:50.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:53:50.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:53:50.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:53:50.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:53:50.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:53:50.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:53:50.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:53:50.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:53:50.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:53:50.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:53:50.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:53:50.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:53:50.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:53:50.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:53:50.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:53:50.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:53:50.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:53:50.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:53:50.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:53:50.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:53:50.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:53:50.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:53:50.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:53:50.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:53:50.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:53:50.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:53:50.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:53:50.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:53:50.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:53:50.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:53:50.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:53:50.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:53:50.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:53:50.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:53:50.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:53:50.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:53:50.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:53:50.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:53:50.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:53:50.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:53:50.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:53:50.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:53:50.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:53:50.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:53:50.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:53:50.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:53:50.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:53:50.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:53:50.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:53:50.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:53:50.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:53:50.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:53:50.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:53:50.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: main() 2024-01-13T11:53:50.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:53:50.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:53:50.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:53:50.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:53:50.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:53:50.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:53:50.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:53:50.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:53:50.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:53:50.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:53:50.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:53:50.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:53:50.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:53:50.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:53:50.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpopuohfub:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpx0l9j9xt:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:53:50.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: audit 2024-01-13T11:53:49.098438+0000 mon.smithi088 (mon.0) 453 : audit [INF] from='client.? 172.21.15.88:0/19440239' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a569dabb-4b1a-4079-bb80-78f2197d03d7"}]: dispatch 2024-01-13T11:53:50.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: audit 2024-01-13T11:53:49.104434+0000 mon.smithi088 (mon.0) 454 : audit [INF] from='client.? 172.21.15.88:0/19440239' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a569dabb-4b1a-4079-bb80-78f2197d03d7"}]': finished 2024-01-13T11:53:50.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: cluster 2024-01-13T11:53:49.104496+0000 mon.smithi088 (mon.0) 455 : cluster [DBG] osdmap e28: 1 total, 0 up, 1 in 2024-01-13T11:53:50.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: audit 2024-01-13T11:53:49.104730+0000 mon.smithi088 (mon.0) 456 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:53:50.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: audit 2024-01-13T11:53:49.443240+0000 mon.smithi088 (mon.0) 457 : audit [INF] from='client.? 172.21.15.88:0/3095493528' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:53:50.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: audit 2024-01-13T11:53:49.448472+0000 mon.smithi088 (mon.0) 458 : audit [INF] from='client.? 172.21.15.88:0/3095493528' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:53:50.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:49 smithi138 bash[17620]: cluster 2024-01-13T11:53:49.448535+0000 mon.smithi088 (mon.0) 459 : cluster [DBG] osdmap e29: 0 total, 0 up, 0 in 2024-01-13T11:53:50.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: cluster 2024-01-13T11:53:48.426875+0000 mgr.smithi088.xwlxdh (mgr.14182) 225 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:50.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: cephadm 2024-01-13T11:53:48.452917+0000 mgr.smithi088.xwlxdh (mgr.14182) 226 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:53:50.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:53:50.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:53:50.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: return f(*arg) 2024-01-13T11:53:50.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:53:50.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:53:50.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:53:50.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: code, '\n'.join(err))) 2024-01-13T11:53:50.392 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:53:50.392 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpopuohfub:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpx0l9j9xt:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:53:50.392 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:53:50.392 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:53:50.392 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:53:50.392 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 24553668-7d79-4dfd-bd47-7eebe1d6b96b 2024-01-13T11:53:50.392 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:53:50.392 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:53:50.392 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:53:50.392 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:53:50.392 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:53:50.392 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:53:50.392 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:53:50.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:53:50.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:53:50.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:53:50.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:53:50.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:53:50.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:53:50.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:53:50.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:53:50.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:53:50.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:53:50.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:53:50.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:53:50.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:53:50.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:53:50.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:53:50.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:53:50.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:53:50.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:53:50.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:53:50.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:53:50.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:53:50.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:53:50.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:53:50.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:53:50.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:53:50.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:53:50.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:53:50.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:53:50.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:53:50.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:53:50.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:53:50.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:53:50.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:53:50.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:53:50.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:53:50.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:53:50.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:53:50.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:53:50.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:53:50.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:53:50.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:53:50.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:53:50.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:53:50.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:53:50.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:53:50.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:53:50.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:53:50.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:53:50.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:53:50.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:53:50.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:53:50.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: main() 2024-01-13T11:53:50.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:53:50.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:53:50.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:53:50.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:53:50.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:53:50.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:53:50.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:53:50.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:53:50.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:53:50.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:53:50.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:53:50.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:53:50.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:53:50.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:53:50.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpopuohfub:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpx0l9j9xt:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:53:50.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: audit 2024-01-13T11:53:49.098438+0000 mon.smithi088 (mon.0) 453 : audit [INF] from='client.? 172.21.15.88:0/19440239' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a569dabb-4b1a-4079-bb80-78f2197d03d7"}]: dispatch 2024-01-13T11:53:50.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: audit 2024-01-13T11:53:49.104434+0000 mon.smithi088 (mon.0) 454 : audit [INF] from='client.? 172.21.15.88:0/19440239' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a569dabb-4b1a-4079-bb80-78f2197d03d7"}]': finished 2024-01-13T11:53:50.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: cluster 2024-01-13T11:53:49.104496+0000 mon.smithi088 (mon.0) 455 : cluster [DBG] osdmap e28: 1 total, 0 up, 1 in 2024-01-13T11:53:50.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: audit 2024-01-13T11:53:49.104730+0000 mon.smithi088 (mon.0) 456 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:53:50.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: audit 2024-01-13T11:53:49.443240+0000 mon.smithi088 (mon.0) 457 : audit [INF] from='client.? 172.21.15.88:0/3095493528' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:53:50.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: audit 2024-01-13T11:53:49.448472+0000 mon.smithi088 (mon.0) 458 : audit [INF] from='client.? 172.21.15.88:0/3095493528' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:53:50.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:49 smithi088 bash[14069]: cluster 2024-01-13T11:53:49.448535+0000 mon.smithi088 (mon.0) 459 : cluster [DBG] osdmap e29: 0 total, 0 up, 0 in 2024-01-13T11:53:51.121 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:50 smithi088 bash[14069]: cluster 2024-01-13T11:53:50.924499+0000 mon.smithi088 (mon.0) 460 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T11:53:51.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:50 smithi138 bash[17620]: cluster 2024-01-13T11:53:50.924499+0000 mon.smithi088 (mon.0) 460 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T11:53:52.056 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:53:52.325 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: cluster 2024-01-13T11:53:50.427200+0000 mgr.smithi088.xwlxdh (mgr.14182) 227 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:52.326 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: cephadm 2024-01-13T11:53:50.777997+0000 mgr.smithi088.xwlxdh (mgr.14182) 228 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:53:52.326 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:53:52.326 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:53:52.326 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: return f(*arg) 2024-01-13T11:53:52.326 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:53:52.326 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:53:52.326 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:53:52.326 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: code, '\n'.join(err))) 2024-01-13T11:53:52.326 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi088/config 2024-01-13T11:53:52.326 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp3zvbqvx4:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpvmzmx8rk:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:53:52.326 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:53:52.327 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:53:52.327 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:53:52.327 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new a569dabb-4b1a-4079-bb80-78f2197d03d7 2024-01-13T11:53:52.327 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:53:52.327 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:53:52.327 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:53:52.327 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:53:52.327 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:53:52.327 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:53:52.327 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:53:52.327 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:53:52.327 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:53:52.327 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:53:52.327 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:53:52.328 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:53:52.328 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:53:52.328 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:53:52.328 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:53:52.328 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:53:52.328 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:53:52.328 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:53:52.328 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:53:52.328 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:53:52.328 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:53:52.328 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:53:52.328 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:53:52.328 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:53:52.329 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:53:52.329 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:53:52.329 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:53:52.329 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:53:52.329 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:53:52.329 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:53:52.329 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:53:52.329 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:53:52.329 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:53:52.329 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:53:52.329 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:53:52.329 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:53:52.329 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:53:52.329 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:53:52.329 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:53:52.331 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:53:52.331 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:53:52.331 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:53:52.331 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:53:52.331 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:53:52.331 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:53:52.331 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:53:52.332 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:53:52.332 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:53:52.332 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:53:52.332 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:53:52.332 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:53:52.332 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:53:52.332 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:53:52.332 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:53:52.332 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:53:52.332 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:53:52.332 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:53:52.332 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:53:52.332 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: main() 2024-01-13T11:53:52.333 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:53:52.333 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:53:52.333 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:53:52.333 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:53:52.333 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:53:52.333 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:53:52.333 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:53:52.333 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:53:52.333 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:53:52.333 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:53:52.333 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:53:52.333 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:53:52.333 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:53:52.333 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:53:52.333 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp3zvbqvx4:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpvmzmx8rk:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:53:52.334 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: cephadm 2024-01-13T11:53:50.780114+0000 mgr.smithi088.xwlxdh (mgr.14182) 229 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T11:53:52.334 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: service_id: all-available-devices 2024-01-13T11:53:52.334 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: service_name: osd.all-available-devices 2024-01-13T11:53:52.334 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: placement: 2024-01-13T11:53:52.334 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: host_pattern: '*' 2024-01-13T11:53:52.334 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: spec: 2024-01-13T11:53:52.334 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: data_devices: 2024-01-13T11:53:52.334 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: all: true 2024-01-13T11:53:52.334 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: filter_logic: AND 2024-01-13T11:53:52.334 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: objectstore: bluestore 2024-01-13T11:53:52.334 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:53:52.334 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpopuohfub:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpx0l9j9xt:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:53:52.334 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:53:52.335 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:53:52.335 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:53:52.335 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 24553668-7d79-4dfd-bd47-7eebe1d6b96b 2024-01-13T11:53:52.335 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:53:52.335 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:53:52.335 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:53:52.335 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:53:52.335 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:53:52.335 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:53:52.335 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:53:52.335 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:53:52.335 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:53:52.335 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:53:52.335 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:53:52.336 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:53:52.336 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:53:52.336 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:53:52.336 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:53:52.336 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:53:52.336 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:53:52.336 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:53:52.336 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:53:52.336 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:53:52.336 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:53:52.336 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:53:52.336 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:53:52.336 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:53:52.336 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:53:52.336 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:53:52.337 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:53:52.337 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:53:52.337 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:53:52.337 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:53:52.337 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:53:52.337 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:53:52.337 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:53:52.337 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:53:52.337 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:53:52.337 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:53:52.337 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:53:52.337 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:53:52.337 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:53:52.337 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:53:52.337 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:53:52.338 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:53:52.338 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:53:52.338 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:53:52.338 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:53:52.338 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:53:52.338 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:53:52.338 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:53:52.338 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:53:52.338 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:53:52.338 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:53:52.338 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:53:52.338 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:53:52.338 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:53:52.338 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:53:52.339 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:53:52.339 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:53:52.339 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:53:52.339 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: main() 2024-01-13T11:53:52.339 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:53:52.339 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:53:52.339 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:53:52.339 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:53:52.339 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:53:52.339 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:53:52.339 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:53:52.339 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:53:52.339 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:53:52.339 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:53:52.340 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:53:52.340 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:53:52.340 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:53:52.340 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:53:52.340 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpopuohfub:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpx0l9j9xt:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:53:52.340 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:53:52.340 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T11:53:52.340 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: if self._apply_service(spec): 2024-01-13T11:53:52.340 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T11:53:52.340 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T11:53:52.340 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T11:53:52.340 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T11:53:52.340 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T11:53:52.341 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T11:53:52.341 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T11:53:52.341 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T11:53:52.341 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T11:53:52.341 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: raise self._value 2024-01-13T11:53:52.341 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T11:53:52.341 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: result = (True, func(*args, **kwds)) 2024-01-13T11:53:52.341 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T11:53:52.341 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: return list(map(*args)) 2024-01-13T11:53:52.341 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:53:52.341 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: return f(*arg) 2024-01-13T11:53:52.341 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:53:52.341 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:53:52.341 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:53:52.342 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: code, '\n'.join(err))) 2024-01-13T11:53:52.342 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:53:52.342 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpopuohfub:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpx0l9j9xt:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:53:52.342 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:53:52.342 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:53:52.342 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:53:52.342 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 24553668-7d79-4dfd-bd47-7eebe1d6b96b 2024-01-13T11:53:52.342 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:53:52.342 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:53:52.342 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:53:52.342 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:53:52.342 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:53:52.342 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:53:52.343 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:53:52.343 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:53:52.343 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:53:52.343 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:53:52.343 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:53:52.343 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:53:52.343 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:53:52.343 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:53:52.343 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:53:52.343 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:53:52.343 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:53:52.343 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:53:52.344 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:53:52.344 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:53:52.344 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:53:52.344 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:53:52.344 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:53:52.344 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:53:52.344 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:53:52.344 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:53:52.344 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:53:52.344 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:53:52.344 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:53:52.344 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:53:52.344 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:53:52.345 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:53:52.345 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:53:52.345 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:53:52.345 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:53:52.345 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:53:52.345 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:53:52.345 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:53:52.345 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:53:52.345 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:53:52.345 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:53:52.345 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:53:52.345 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:53:52.345 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:53:52.345 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:53:52.346 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:53:52.346 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:53:52.346 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:53:52.346 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:53:52.346 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:53:52.346 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:53:52.346 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:53:52.346 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:53:52.346 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:53:52.346 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:53:52.346 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:53:52.346 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:53:52.346 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:53:52.346 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: main() 2024-01-13T11:53:52.346 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:53:52.347 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:53:52.347 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:53:52.347 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:53:52.347 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:53:52.347 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:53:52.347 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:53:52.347 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:53:52.347 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:53:52.347 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:53:52.347 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:53:52.347 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:53:52.347 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:53:52.347 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:53:52.348 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpopuohfub:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpx0l9j9xt:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:53:52.348 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:51 smithi088 bash[14069]: cluster 2024-01-13T11:53:50.780847+0000 mgr.smithi088.xwlxdh (mgr.14182) 230 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:52.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: cluster 2024-01-13T11:53:50.427200+0000 mgr.smithi088.xwlxdh (mgr.14182) 227 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:52.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: cephadm 2024-01-13T11:53:50.777997+0000 mgr.smithi088.xwlxdh (mgr.14182) 228 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:53:52.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:53:52.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:53:52.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: return f(*arg) 2024-01-13T11:53:52.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:53:52.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:53:52.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:53:52.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: code, '\n'.join(err))) 2024-01-13T11:53:52.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi088/config 2024-01-13T11:53:52.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp3zvbqvx4:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpvmzmx8rk:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:53:52.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:53:52.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:53:52.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:53:52.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new a569dabb-4b1a-4079-bb80-78f2197d03d7 2024-01-13T11:53:52.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:53:52.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:53:52.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:53:52.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:53:52.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:53:52.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:53:52.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:53:52.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:53:52.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:53:52.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:53:52.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:53:52.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:53:52.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:53:52.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:53:52.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:53:52.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:53:52.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:53:52.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:53:52.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:53:52.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:53:52.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:53:52.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:53:52.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:53:52.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:53:52.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:53:52.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:53:52.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:53:52.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:53:52.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:53:52.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:53:52.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:53:52.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:53:52.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:53:52.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:53:52.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:53:52.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:53:52.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:53:52.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:53:52.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:53:52.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:53:52.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:53:52.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:53:52.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:53:52.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:53:52.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:53:52.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:53:52.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:53:52.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:53:52.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:53:52.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:53:52.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:53:52.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:53:52.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:53:52.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:53:52.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:53:52.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:53:52.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:53:52.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:53:52.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: main() 2024-01-13T11:53:52.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:53:52.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:53:52.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:53:52.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:53:52.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:53:52.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:53:52.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:53:52.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:53:52.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:53:52.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:53:52.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:53:52.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:53:52.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:53:52.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:53:52.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp3zvbqvx4:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpvmzmx8rk:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:53:52.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: cephadm 2024-01-13T11:53:50.780114+0000 mgr.smithi088.xwlxdh (mgr.14182) 229 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T11:53:52.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: service_id: all-available-devices 2024-01-13T11:53:52.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: service_name: osd.all-available-devices 2024-01-13T11:53:52.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: placement: 2024-01-13T11:53:52.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: host_pattern: '*' 2024-01-13T11:53:52.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: spec: 2024-01-13T11:53:52.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: data_devices: 2024-01-13T11:53:52.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: all: true 2024-01-13T11:53:52.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: filter_logic: AND 2024-01-13T11:53:52.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: objectstore: bluestore 2024-01-13T11:53:52.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:53:52.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpopuohfub:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpx0l9j9xt:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:53:52.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:53:52.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:53:52.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:53:52.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 24553668-7d79-4dfd-bd47-7eebe1d6b96b 2024-01-13T11:53:52.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:53:52.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:53:52.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:53:52.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:53:52.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:53:52.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:53:52.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:53:52.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:53:52.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:53:52.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:53:52.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:53:52.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:53:52.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:53:52.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:53:52.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:53:52.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:53:52.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:53:52.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:53:52.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:53:52.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:53:52.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:53:52.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:53:52.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:53:52.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:53:52.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:53:52.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:53:52.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:53:52.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:53:52.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:53:52.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:51 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:53:52.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:53:52.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:53:52.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:53:52.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:53:52.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:53:52.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:53:52.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:53:52.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:53:52.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:53:52.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:53:52.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:53:52.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:53:52.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:53:52.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:53:52.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:53:52.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:53:52.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:53:52.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:53:52.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:53:52.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:53:52.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:53:52.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:53:52.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:53:52.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:53:52.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:53:52.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:53:52.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:53:52.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:53:52.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: main() 2024-01-13T11:53:52.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:53:52.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:53:52.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:53:52.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:53:52.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:53:52.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:53:52.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:53:52.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:53:52.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:53:52.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:53:52.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:53:52.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:53:52.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:53:52.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:53:52.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpopuohfub:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpx0l9j9xt:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:53:52.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:53:52.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T11:53:52.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: if self._apply_service(spec): 2024-01-13T11:53:52.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T11:53:52.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T11:53:52.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T11:53:52.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T11:53:52.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T11:53:52.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T11:53:52.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T11:53:52.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T11:53:52.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T11:53:52.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: raise self._value 2024-01-13T11:53:52.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T11:53:52.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: result = (True, func(*args, **kwds)) 2024-01-13T11:53:52.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T11:53:52.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: return list(map(*args)) 2024-01-13T11:53:52.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:53:52.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: return f(*arg) 2024-01-13T11:53:52.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:53:52.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:53:52.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:53:52.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: code, '\n'.join(err))) 2024-01-13T11:53:52.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:53:52.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpopuohfub:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpx0l9j9xt:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:53:52.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:53:52.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:53:52.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:53:52.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 24553668-7d79-4dfd-bd47-7eebe1d6b96b 2024-01-13T11:53:52.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:53:52.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:53:52.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:53:52.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:53:52.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:53:52.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:53:52.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:53:52.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:53:52.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:53:52.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:53:52.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:53:52.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:53:52.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:53:52.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:53:52.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:53:52.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:53:52.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:53:52.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:53:52.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:53:52.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:53:52.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:53:52.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:53:52.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:53:52.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:53:52.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:53:52.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:53:52.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:53:52.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:53:52.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:53:52.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:53:52.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:53:52.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:53:52.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:53:52.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:53:52.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:53:52.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:53:52.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:53:52.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:53:52.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:53:52.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:53:52.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:53:52.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:53:52.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:53:52.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:53:52.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:53:52.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:53:52.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:53:52.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:53:52.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:53:52.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:53:52.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:53:52.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:53:52.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:53:52.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:53:52.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:53:52.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:53:52.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:53:52.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:53:52.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: main() 2024-01-13T11:53:52.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:53:52.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:53:52.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:53:52.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:53:52.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:53:52.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:53:52.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:53:52.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:53:52.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:53:52.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:53:52.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:53:52.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:53:52.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:53:52.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:53:52.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpopuohfub:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpx0l9j9xt:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:53:52.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:52 smithi138 bash[17620]: cluster 2024-01-13T11:53:50.780847+0000 mgr.smithi088.xwlxdh (mgr.14182) 230 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:52.850 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:53:53.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:52 smithi088 bash[14069]: audit 2024-01-13T11:53:52.067952+0000 mon.smithi088 (mon.0) 461 : audit [DBG] from='client.? 172.21.15.88:0/2649425338' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:53:53.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:52 smithi088 bash[14069]: cluster 2024-01-13T11:53:52.782097+0000 mgr.smithi088.xwlxdh (mgr.14182) 231 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:53.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:53 smithi138 bash[17620]: audit 2024-01-13T11:53:52.067952+0000 mon.smithi088 (mon.0) 461 : audit [DBG] from='client.? 172.21.15.88:0/2649425338' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:53:53.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:53 smithi138 bash[17620]: cluster 2024-01-13T11:53:52.782097+0000 mgr.smithi088.xwlxdh (mgr.14182) 231 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:53.851 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:53:56.123 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:55 smithi138 bash[17620]: cluster 2024-01-13T11:53:54.782968+0000 mgr.smithi088.xwlxdh (mgr.14182) 232 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:56.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:55 smithi088 bash[14069]: cluster 2024-01-13T11:53:54.782968+0000 mgr.smithi088.xwlxdh (mgr.14182) 232 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:57.126 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:53:57.901 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:53:58.123 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:57 smithi138 bash[17620]: cluster 2024-01-13T11:53:56.783623+0000 mgr.smithi088.xwlxdh (mgr.14182) 233 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:58.123 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:57 smithi138 bash[17620]: audit 2024-01-13T11:53:57.137167+0000 mon.smithi088 (mon.0) 462 : audit [DBG] from='client.? 172.21.15.88:0/3225912806' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:53:58.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:57 smithi088 bash[14069]: cluster 2024-01-13T11:53:56.783623+0000 mgr.smithi088.xwlxdh (mgr.14182) 233 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:53:58.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:57 smithi088 bash[14069]: audit 2024-01-13T11:53:57.137167+0000 mon.smithi088 (mon.0) 462 : audit [DBG] from='client.? 172.21.15.88:0/3225912806' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:53:58.902 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:54:00.175 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:59 smithi088 bash[14069]: cluster 2024-01-13T11:53:58.784671+0000 mgr.smithi088.xwlxdh (mgr.14182) 234 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:00.175 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:53:59 smithi088 bash[14069]: audit 2024-01-13T11:53:59.733087+0000 mon.smithi088 (mon.0) 463 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:54:00.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:59 smithi138 bash[17620]: cluster 2024-01-13T11:53:58.784671+0000 mgr.smithi088.xwlxdh (mgr.14182) 234 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:00.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:53:59 smithi138 bash[17620]: audit 2024-01-13T11:53:59.733087+0000 mon.smithi088 (mon.0) 463 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:54:02.035 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:54:02.288 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:01 smithi088 bash[14069]: cluster 2024-01-13T11:54:00.785743+0000 mgr.smithi088.xwlxdh (mgr.14182) 235 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:02.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:01 smithi138 bash[17620]: cluster 2024-01-13T11:54:00.785743+0000 mgr.smithi088.xwlxdh (mgr.14182) 235 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:02.791 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:54:03.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:02 smithi088 bash[14069]: audit 2024-01-13T11:54:02.046123+0000 mon.smithi088 (mon.0) 464 : audit [DBG] from='client.? 172.21.15.88:0/3772922273' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:54:03.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:02 smithi138 bash[17620]: audit 2024-01-13T11:54:02.046123+0000 mon.smithi088 (mon.0) 464 : audit [DBG] from='client.? 172.21.15.88:0/3772922273' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:54:03.792 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:54:04.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:03 smithi088 bash[14069]: cluster 2024-01-13T11:54:02.786665+0000 mgr.smithi088.xwlxdh (mgr.14182) 236 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:04.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:03 smithi138 bash[17620]: cluster 2024-01-13T11:54:02.786665+0000 mgr.smithi088.xwlxdh (mgr.14182) 236 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:06.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:05 smithi088 bash[14069]: cluster 2024-01-13T11:54:04.787524+0000 mgr.smithi088.xwlxdh (mgr.14182) 237 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:06.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:05 smithi138 bash[17620]: cluster 2024-01-13T11:54:04.787524+0000 mgr.smithi088.xwlxdh (mgr.14182) 237 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:07.112 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:54:07.908 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:54:08.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:07 smithi138 bash[17620]: cluster 2024-01-13T11:54:06.788418+0000 mgr.smithi088.xwlxdh (mgr.14182) 238 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:08.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:07 smithi138 bash[17620]: audit 2024-01-13T11:54:07.123353+0000 mon.smithi088 (mon.0) 465 : audit [DBG] from='client.? 172.21.15.88:0/2417449848' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:54:08.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:07 smithi088 bash[14069]: cluster 2024-01-13T11:54:06.788418+0000 mgr.smithi088.xwlxdh (mgr.14182) 238 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:08.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:07 smithi088 bash[14069]: audit 2024-01-13T11:54:07.123353+0000 mon.smithi088 (mon.0) 465 : audit [DBG] from='client.? 172.21.15.88:0/2417449848' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:54:08.908 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:54:10.370 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:09 smithi088 bash[14069]: cluster 2024-01-13T11:54:08.789413+0000 mgr.smithi088.xwlxdh (mgr.14182) 239 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:10.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:09 smithi138 bash[17620]: cluster 2024-01-13T11:54:08.789413+0000 mgr.smithi088.xwlxdh (mgr.14182) 239 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:12.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:11 smithi088 bash[14069]: cluster 2024-01-13T11:54:10.790404+0000 mgr.smithi088.xwlxdh (mgr.14182) 240 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:12.241 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:54:12.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:11 smithi138 bash[17620]: cluster 2024-01-13T11:54:10.790404+0000 mgr.smithi088.xwlxdh (mgr.14182) 240 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:13.089 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:54:13.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:12 smithi138 bash[17620]: audit 2024-01-13T11:54:12.252484+0000 mon.smithi088 (mon.0) 466 : audit [DBG] from='client.? 172.21.15.88:0/2874010918' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:54:13.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:12 smithi088 bash[14069]: audit 2024-01-13T11:54:12.252484+0000 mon.smithi088 (mon.0) 466 : audit [DBG] from='client.? 172.21.15.88:0/2874010918' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:54:14.090 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:54:14.286 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:14 smithi088 bash[14069]: cluster 2024-01-13T11:54:12.791424+0000 mgr.smithi088.xwlxdh (mgr.14182) 241 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:14.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:14 smithi138 bash[17620]: cluster 2024-01-13T11:54:12.791424+0000 mgr.smithi088.xwlxdh (mgr.14182) 241 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:15.368 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:15 smithi088 bash[14069]: cluster 2024-01-13T11:54:14.792412+0000 mgr.smithi088.xwlxdh (mgr.14182) 242 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:15.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:15 smithi138 bash[17620]: cluster 2024-01-13T11:54:14.792412+0000 mgr.smithi088.xwlxdh (mgr.14182) 242 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:17.327 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:54:18.123 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:17 smithi138 bash[17620]: cluster 2024-01-13T11:54:16.793403+0000 mgr.smithi088.xwlxdh (mgr.14182) 243 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:18.123 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:17 smithi138 bash[17620]: audit 2024-01-13T11:54:17.338171+0000 mon.smithi088 (mon.0) 467 : audit [DBG] from='client.? 172.21.15.88:0/1997993362' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:54:18.124 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:17 smithi088 bash[14069]: cluster 2024-01-13T11:54:16.793403+0000 mgr.smithi088.xwlxdh (mgr.14182) 243 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:18.124 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:17 smithi088 bash[14069]: audit 2024-01-13T11:54:17.338171+0000 mon.smithi088 (mon.0) 467 : audit [DBG] from='client.? 172.21.15.88:0/1997993362' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:54:18.125 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:54:19.127 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:54:20.218 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:19 smithi088 bash[14069]: cluster 2024-01-13T11:54:18.794456+0000 mgr.smithi088.xwlxdh (mgr.14182) 244 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:20.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:19 smithi138 bash[17620]: cluster 2024-01-13T11:54:18.794456+0000 mgr.smithi088.xwlxdh (mgr.14182) 244 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:22.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:21 smithi138 bash[17620]: cluster 2024-01-13T11:54:20.795005+0000 mgr.smithi088.xwlxdh (mgr.14182) 245 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:22.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:21 smithi088 bash[14069]: cluster 2024-01-13T11:54:20.795005+0000 mgr.smithi088.xwlxdh (mgr.14182) 245 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:22.421 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:54:23.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:22 smithi088 bash[14069]: audit 2024-01-13T11:54:22.432391+0000 mon.smithi088 (mon.0) 468 : audit [DBG] from='client.? 172.21.15.88:0/900100626' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:54:23.226 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:54:23.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:22 smithi138 bash[17620]: audit 2024-01-13T11:54:22.432391+0000 mon.smithi088 (mon.0) 468 : audit [DBG] from='client.? 172.21.15.88:0/900100626' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:54:24.228 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:54:24.243 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:23 smithi088 bash[14069]: cluster 2024-01-13T11:54:22.795606+0000 mgr.smithi088.xwlxdh (mgr.14182) 246 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:24.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:23 smithi138 bash[17620]: cluster 2024-01-13T11:54:22.795606+0000 mgr.smithi088.xwlxdh (mgr.14182) 246 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:26.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:25 smithi138 bash[17620]: cluster 2024-01-13T11:54:24.796463+0000 mgr.smithi088.xwlxdh (mgr.14182) 247 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:26.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:25 smithi088 bash[14069]: cluster 2024-01-13T11:54:24.796463+0000 mgr.smithi088.xwlxdh (mgr.14182) 247 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:27.275 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:54:28.084 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:54:28.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:27 smithi138 bash[17620]: cluster 2024-01-13T11:54:26.797387+0000 mgr.smithi088.xwlxdh (mgr.14182) 248 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:28.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:27 smithi138 bash[17620]: audit 2024-01-13T11:54:27.286808+0000 mon.smithi088 (mon.0) 469 : audit [DBG] from='client.? 172.21.15.88:0/556777823' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:54:28.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:27 smithi088 bash[14069]: cluster 2024-01-13T11:54:26.797387+0000 mgr.smithi088.xwlxdh (mgr.14182) 248 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:28.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:27 smithi088 bash[14069]: audit 2024-01-13T11:54:27.286808+0000 mon.smithi088 (mon.0) 469 : audit [DBG] from='client.? 172.21.15.88:0/556777823' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:54:29.085 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:54:30.288 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:29 smithi088 bash[14069]: cluster 2024-01-13T11:54:28.798423+0000 mgr.smithi088.xwlxdh (mgr.14182) 249 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:30.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:29 smithi138 bash[17620]: cluster 2024-01-13T11:54:28.798423+0000 mgr.smithi088.xwlxdh (mgr.14182) 249 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:32.295 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:54:32.325 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:31 smithi088 bash[14069]: cluster 2024-01-13T11:54:30.799447+0000 mgr.smithi088.xwlxdh (mgr.14182) 250 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:32.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:31 smithi138 bash[17620]: cluster 2024-01-13T11:54:30.799447+0000 mgr.smithi088.xwlxdh (mgr.14182) 250 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:33.101 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:54:33.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:33 smithi138 bash[17620]: audit 2024-01-13T11:54:32.306325+0000 mon.smithi088 (mon.0) 470 : audit [DBG] from='client.? 172.21.15.88:0/3138090328' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:54:33.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:33 smithi138 bash[17620]: cluster 2024-01-13T11:54:32.800518+0000 mgr.smithi088.xwlxdh (mgr.14182) 251 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:33.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:33 smithi088 bash[14069]: audit 2024-01-13T11:54:32.306325+0000 mon.smithi088 (mon.0) 470 : audit [DBG] from='client.? 172.21.15.88:0/3138090328' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:54:33.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:33 smithi088 bash[14069]: cluster 2024-01-13T11:54:32.800518+0000 mgr.smithi088.xwlxdh (mgr.14182) 251 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:34.102 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:54:36.123 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:35 smithi138 bash[17620]: cluster 2024-01-13T11:54:34.801594+0000 mgr.smithi088.xwlxdh (mgr.14182) 252 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:36.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:35 smithi088 bash[14069]: cluster 2024-01-13T11:54:34.801594+0000 mgr.smithi088.xwlxdh (mgr.14182) 252 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:37.449 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:54:38.123 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:37 smithi138 bash[17620]: cluster 2024-01-13T11:54:36.802375+0000 mgr.smithi088.xwlxdh (mgr.14182) 253 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:38.123 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:37 smithi138 bash[17620]: audit 2024-01-13T11:54:37.461015+0000 mon.smithi088 (mon.0) 471 : audit [DBG] from='client.? 172.21.15.88:0/2393442472' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:54:38.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:37 smithi088 bash[14069]: cluster 2024-01-13T11:54:36.802375+0000 mgr.smithi088.xwlxdh (mgr.14182) 253 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:38.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:37 smithi088 bash[14069]: audit 2024-01-13T11:54:37.461015+0000 mon.smithi088 (mon.0) 471 : audit [DBG] from='client.? 172.21.15.88:0/2393442472' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:54:38.221 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:54:39.222 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:54:40.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:39 smithi138 bash[17620]: cluster 2024-01-13T11:54:38.803425+0000 mgr.smithi088.xwlxdh (mgr.14182) 254 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:40.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:39 smithi088 bash[14069]: cluster 2024-01-13T11:54:38.803425+0000 mgr.smithi088.xwlxdh (mgr.14182) 254 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:42.216 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:41 smithi088 bash[14069]: cluster 2024-01-13T11:54:40.804161+0000 mgr.smithi088.xwlxdh (mgr.14182) 255 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:42.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:41 smithi138 bash[17620]: cluster 2024-01-13T11:54:40.804161+0000 mgr.smithi088.xwlxdh (mgr.14182) 255 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:42.594 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:54:43.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:42 smithi088 bash[14069]: audit 2024-01-13T11:54:42.605436+0000 mon.smithi088 (mon.0) 472 : audit [DBG] from='client.? 172.21.15.88:0/2331779446' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:54:43.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:42 smithi138 bash[17620]: audit 2024-01-13T11:54:42.605436+0000 mon.smithi088 (mon.0) 472 : audit [DBG] from='client.? 172.21.15.88:0/2331779446' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:54:43.398 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:54:44.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:43 smithi138 bash[17620]: cluster 2024-01-13T11:54:42.805131+0000 mgr.smithi088.xwlxdh (mgr.14182) 256 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:44.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:43 smithi088 bash[14069]: cluster 2024-01-13T11:54:42.805131+0000 mgr.smithi088.xwlxdh (mgr.14182) 256 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:44.399 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:54:46.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:45 smithi138 bash[17620]: cluster 2024-01-13T11:54:44.806232+0000 mgr.smithi088.xwlxdh (mgr.14182) 257 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:46.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:45 smithi088 bash[14069]: cluster 2024-01-13T11:54:44.806232+0000 mgr.smithi088.xwlxdh (mgr.14182) 257 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:47.707 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:54:48.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:47 smithi088 bash[14069]: cluster 2024-01-13T11:54:46.807303+0000 mgr.smithi088.xwlxdh (mgr.14182) 258 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:48.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:47 smithi088 bash[14069]: audit 2024-01-13T11:54:47.718663+0000 mon.smithi088 (mon.0) 473 : audit [DBG] from='client.? 172.21.15.88:0/3873848767' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:54:48.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:47 smithi138 bash[17620]: cluster 2024-01-13T11:54:46.807303+0000 mgr.smithi088.xwlxdh (mgr.14182) 258 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:48.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:47 smithi138 bash[17620]: audit 2024-01-13T11:54:47.718663+0000 mon.smithi088 (mon.0) 473 : audit [DBG] from='client.? 172.21.15.88:0/3873848767' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:54:48.522 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:54:49.523 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:54:50.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:49 smithi138 bash[17620]: cluster 2024-01-13T11:54:48.808443+0000 mgr.smithi088.xwlxdh (mgr.14182) 259 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:50.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:49 smithi088 bash[14069]: cluster 2024-01-13T11:54:48.808443+0000 mgr.smithi088.xwlxdh (mgr.14182) 259 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:51.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:51 smithi138 bash[17620]: audit 2024-01-13T11:54:50.782699+0000 mon.smithi088 (mon.0) 474 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:54:51.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:51 smithi088 bash[14069]: audit 2024-01-13T11:54:50.782699+0000 mon.smithi088 (mon.0) 474 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:54:52.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:52 smithi138 bash[17620]: cluster 2024-01-13T11:54:50.809162+0000 mgr.smithi088.xwlxdh (mgr.14182) 260 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:52.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:52 smithi138 bash[17620]: audit 2024-01-13T11:54:51.080291+0000 mon.smithi088 (mon.0) 475 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:54:52.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:52 smithi138 bash[17620]: audit 2024-01-13T11:54:51.081548+0000 mon.smithi088 (mon.0) 476 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T11:54:52.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:52 smithi138 bash[17620]: audit 2024-01-13T11:54:51.089587+0000 mon.smithi088 (mon.0) 477 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:54:52.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:52 smithi138 bash[17620]: cluster 2024-01-13T11:54:51.090142+0000 mgr.smithi088.xwlxdh (mgr.14182) 261 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:52.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:52 smithi138 bash[17620]: audit 2024-01-13T11:54:51.095064+0000 mon.smithi088 (mon.0) 478 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T11:54:52.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:52 smithi138 bash[17620]: audit 2024-01-13T11:54:51.099477+0000 mon.smithi088 (mon.0) 479 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:54:52.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:52 smithi138 bash[17620]: audit 2024-01-13T11:54:51.102466+0000 mon.smithi088 (mon.0) 480 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:54:52.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:52 smithi138 bash[17620]: audit 2024-01-13T11:54:51.102905+0000 mon.smithi088 (mon.0) 481 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:54:52.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:52 smithi138 bash[17620]: audit 2024-01-13T11:54:51.103553+0000 mon.smithi088 (mon.0) 482 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:54:52.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:52 smithi088 bash[14069]: cluster 2024-01-13T11:54:50.809162+0000 mgr.smithi088.xwlxdh (mgr.14182) 260 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:52.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:52 smithi088 bash[14069]: audit 2024-01-13T11:54:51.080291+0000 mon.smithi088 (mon.0) 475 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:54:52.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:52 smithi088 bash[14069]: audit 2024-01-13T11:54:51.081548+0000 mon.smithi088 (mon.0) 476 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T11:54:52.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:52 smithi088 bash[14069]: audit 2024-01-13T11:54:51.089587+0000 mon.smithi088 (mon.0) 477 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:54:52.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:52 smithi088 bash[14069]: cluster 2024-01-13T11:54:51.090142+0000 mgr.smithi088.xwlxdh (mgr.14182) 261 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:52.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:52 smithi088 bash[14069]: audit 2024-01-13T11:54:51.095064+0000 mon.smithi088 (mon.0) 478 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T11:54:52.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:52 smithi088 bash[14069]: audit 2024-01-13T11:54:51.099477+0000 mon.smithi088 (mon.0) 479 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:54:52.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:52 smithi088 bash[14069]: audit 2024-01-13T11:54:51.102466+0000 mon.smithi088 (mon.0) 480 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:54:52.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:52 smithi088 bash[14069]: audit 2024-01-13T11:54:51.102905+0000 mon.smithi088 (mon.0) 481 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:54:52.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:52 smithi088 bash[14069]: audit 2024-01-13T11:54:51.103553+0000 mon.smithi088 (mon.0) 482 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:54:52.872 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:54:53.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:53 smithi138 bash[17620]: cluster 2024-01-13T11:54:52.086107+0000 mon.smithi088 (mon.0) 483 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T11:54:53.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:53 smithi138 bash[17620]: cluster 2024-01-13T11:54:52.086160+0000 mon.smithi088 (mon.0) 484 : cluster [INF] Cluster is now healthy 2024-01-13T11:54:53.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:53 smithi138 bash[17620]: audit 2024-01-13T11:54:52.884114+0000 mon.smithi088 (mon.0) 485 : audit [DBG] from='client.? 172.21.15.88:0/4077363917' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:54:53.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:53 smithi088 bash[14069]: cluster 2024-01-13T11:54:52.086107+0000 mon.smithi088 (mon.0) 483 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T11:54:53.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:53 smithi088 bash[14069]: cluster 2024-01-13T11:54:52.086160+0000 mon.smithi088 (mon.0) 484 : cluster [INF] Cluster is now healthy 2024-01-13T11:54:53.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:53 smithi088 bash[14069]: audit 2024-01-13T11:54:52.884114+0000 mon.smithi088 (mon.0) 485 : audit [DBG] from='client.? 172.21.15.88:0/4077363917' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:54:53.594 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:54:54.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:54 smithi138 bash[17620]: cluster 2024-01-13T11:54:53.090576+0000 mgr.smithi088.xwlxdh (mgr.14182) 262 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:54.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:54 smithi088 bash[14069]: cluster 2024-01-13T11:54:53.090576+0000 mgr.smithi088.xwlxdh (mgr.14182) 262 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:54.595 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:54:56.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:56 smithi088 bash[14069]: cluster 2024-01-13T11:54:55.091045+0000 mgr.smithi088.xwlxdh (mgr.14182) 263 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:56.573 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:56 smithi138 bash[17620]: cluster 2024-01-13T11:54:55.091045+0000 mgr.smithi088.xwlxdh (mgr.14182) 263 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:58.440 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:54:59.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:58 smithi088 bash[14069]: cluster 2024-01-13T11:54:57.091363+0000 mgr.smithi088.xwlxdh (mgr.14182) 264 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:59.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:54:58 smithi088 bash[14069]: audit 2024-01-13T11:54:58.451387+0000 mon.smithi088 (mon.0) 486 : audit [DBG] from='client.? 172.21.15.88:0/398930254' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:54:59.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:58 smithi138 bash[17620]: cluster 2024-01-13T11:54:57.091363+0000 mgr.smithi088.xwlxdh (mgr.14182) 264 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:54:59.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:54:58 smithi138 bash[17620]: audit 2024-01-13T11:54:58.451387+0000 mon.smithi088 (mon.0) 486 : audit [DBG] from='client.? 172.21.15.88:0/398930254' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:54:59.816 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:55:00.818 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:55:00.873 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:00 smithi138 bash[17620]: cluster 2024-01-13T11:54:59.091647+0000 mgr.smithi088.xwlxdh (mgr.14182) 265 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:00.873 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:00 smithi138 bash[17620]: audit 2024-01-13T11:54:59.736082+0000 mon.smithi088 (mon.0) 487 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:55:00.873 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:00 smithi138 bash[17620]: audit 2024-01-13T11:54:59.847082+0000 mon.smithi138 (mon.1) 13 : audit [INF] from='client.? 172.21.15.138:0/3988713972' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "86c693ba-57c1-49eb-8ef0-0f2ecc375b2b"}]: dispatch 2024-01-13T11:55:00.873 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:00 smithi138 bash[17620]: audit 2024-01-13T11:54:59.847531+0000 mon.smithi088 (mon.0) 488 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "86c693ba-57c1-49eb-8ef0-0f2ecc375b2b"}]: dispatch 2024-01-13T11:55:00.873 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:00 smithi138 bash[17620]: audit 2024-01-13T11:54:59.854494+0000 mon.smithi088 (mon.0) 489 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "86c693ba-57c1-49eb-8ef0-0f2ecc375b2b"}]': finished 2024-01-13T11:55:00.873 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:00 smithi138 bash[17620]: cluster 2024-01-13T11:54:59.854607+0000 mon.smithi088 (mon.0) 490 : cluster [DBG] osdmap e30: 1 total, 0 up, 1 in 2024-01-13T11:55:00.874 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:00 smithi138 bash[17620]: audit 2024-01-13T11:54:59.854768+0000 mon.smithi088 (mon.0) 491 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:55:00.874 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:00 smithi138 bash[17620]: audit 2024-01-13T11:55:00.194222+0000 mon.smithi138 (mon.1) 14 : audit [INF] from='client.? 172.21.15.138:0/3151698058' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:55:00.874 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:00 smithi138 bash[17620]: audit 2024-01-13T11:55:00.194869+0000 mon.smithi088 (mon.0) 492 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:55:00.874 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:00 smithi138 bash[17620]: audit 2024-01-13T11:55:00.203586+0000 mon.smithi088 (mon.0) 493 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:55:00.874 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:00 smithi138 bash[17620]: cluster 2024-01-13T11:55:00.203697+0000 mon.smithi088 (mon.0) 494 : cluster [DBG] osdmap e31: 0 total, 0 up, 0 in 2024-01-13T11:55:01.013 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:00 smithi088 bash[14069]: cluster 2024-01-13T11:54:59.091647+0000 mgr.smithi088.xwlxdh (mgr.14182) 265 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:01.013 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:00 smithi088 bash[14069]: audit 2024-01-13T11:54:59.736082+0000 mon.smithi088 (mon.0) 487 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:55:01.013 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:00 smithi088 bash[14069]: audit 2024-01-13T11:54:59.847082+0000 mon.smithi138 (mon.1) 13 : audit [INF] from='client.? 172.21.15.138:0/3988713972' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "86c693ba-57c1-49eb-8ef0-0f2ecc375b2b"}]: dispatch 2024-01-13T11:55:01.013 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:00 smithi088 bash[14069]: audit 2024-01-13T11:54:59.847531+0000 mon.smithi088 (mon.0) 488 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "86c693ba-57c1-49eb-8ef0-0f2ecc375b2b"}]: dispatch 2024-01-13T11:55:01.013 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:00 smithi088 bash[14069]: audit 2024-01-13T11:54:59.854494+0000 mon.smithi088 (mon.0) 489 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "86c693ba-57c1-49eb-8ef0-0f2ecc375b2b"}]': finished 2024-01-13T11:55:01.013 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:00 smithi088 bash[14069]: cluster 2024-01-13T11:54:59.854607+0000 mon.smithi088 (mon.0) 490 : cluster [DBG] osdmap e30: 1 total, 0 up, 1 in 2024-01-13T11:55:01.013 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:00 smithi088 bash[14069]: audit 2024-01-13T11:54:59.854768+0000 mon.smithi088 (mon.0) 491 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:55:01.013 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:00 smithi088 bash[14069]: audit 2024-01-13T11:55:00.194222+0000 mon.smithi138 (mon.1) 14 : audit [INF] from='client.? 172.21.15.138:0/3151698058' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:55:01.013 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:00 smithi088 bash[14069]: audit 2024-01-13T11:55:00.194869+0000 mon.smithi088 (mon.0) 492 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:55:01.014 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:00 smithi088 bash[14069]: audit 2024-01-13T11:55:00.203586+0000 mon.smithi088 (mon.0) 493 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:55:01.014 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:00 smithi088 bash[14069]: cluster 2024-01-13T11:55:00.203697+0000 mon.smithi088 (mon.0) 494 : cluster [DBG] osdmap e31: 0 total, 0 up, 0 in 2024-01-13T11:55:02.873 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: cephadm 2024-01-13T11:55:01.090136+0000 mgr.smithi088.xwlxdh (mgr.14182) 266 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:55:02.874 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:55:02.874 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:55:02.874 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: return f(*arg) 2024-01-13T11:55:02.874 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:55:02.874 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:55:02.875 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:55:02.875 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: code, '\n'.join(err))) 2024-01-13T11:55:02.875 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:55:02.875 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpm4b2cbzw:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmptrxwfmqp:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:55:02.875 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:55:02.876 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:55:02.876 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:55:02.876 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 86c693ba-57c1-49eb-8ef0-0f2ecc375b2b 2024-01-13T11:55:02.876 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:55:02.876 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:55:02.876 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:55:02.877 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:55:02.877 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:55:02.877 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:55:02.877 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:55:02.878 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:55:02.878 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:55:02.878 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:55:02.879 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:55:02.879 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:55:02.879 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:55:02.879 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:55:02.879 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:55:02.879 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:55:02.880 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:55:02.880 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:55:02.880 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:55:02.880 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:55:02.880 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:55:02.881 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:55:02.881 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:55:02.881 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:55:02.881 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:55:02.881 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:55:02.881 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:55:02.882 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:55:02.882 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:55:02.882 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:55:02.882 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:55:02.882 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:55:02.882 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:55:02.883 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:55:02.883 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:55:02.883 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:55:02.883 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:55:02.883 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:55:02.883 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:55:02.883 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:55:02.884 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:55:02.884 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:55:02.884 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:55:02.884 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:55:02.884 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:55:02.885 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:55:02.885 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:55:02.885 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:55:02.885 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:55:02.885 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:55:02.885 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:55:02.886 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:55:02.886 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:55:02.886 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:55:02.886 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:55:02.886 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:55:02.886 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:55:02.886 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:55:02.887 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: main() 2024-01-13T11:55:02.887 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:55:02.887 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:55:02.887 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:55:02.887 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:55:02.888 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:55:02.888 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:55:02.888 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:55:02.888 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:55:02.888 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:55:02.889 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:55:02.889 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:55:02.889 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:55:02.889 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:55:02.889 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:55:02.889 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpm4b2cbzw:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmptrxwfmqp:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:55:02.890 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: cluster 2024-01-13T11:55:01.091879+0000 mgr.smithi088.xwlxdh (mgr.14182) 267 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:02.890 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: audit 2024-01-13T11:55:01.811041+0000 mon.smithi088 (mon.0) 495 : audit [INF] from='client.? 172.21.15.88:0/3963089486' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17643570-e67e-407b-a20a-0fd649501123"}]: dispatch 2024-01-13T11:55:02.890 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: audit 2024-01-13T11:55:01.816814+0000 mon.smithi088 (mon.0) 496 : audit [INF] from='client.? 172.21.15.88:0/3963089486' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "17643570-e67e-407b-a20a-0fd649501123"}]': finished 2024-01-13T11:55:02.890 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: cluster 2024-01-13T11:55:01.816897+0000 mon.smithi088 (mon.0) 497 : cluster [DBG] osdmap e32: 1 total, 0 up, 1 in 2024-01-13T11:55:02.890 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: audit 2024-01-13T11:55:01.817187+0000 mon.smithi088 (mon.0) 498 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:55:02.891 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: audit 2024-01-13T11:55:02.170176+0000 mon.smithi088 (mon.0) 499 : audit [INF] from='client.? 172.21.15.88:0/2075864320' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:55:02.891 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: audit 2024-01-13T11:55:02.175419+0000 mon.smithi088 (mon.0) 500 : audit [INF] from='client.? 172.21.15.88:0/2075864320' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:55:02.891 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:02 smithi138 bash[17620]: cluster 2024-01-13T11:55:02.175484+0000 mon.smithi088 (mon.0) 501 : cluster [DBG] osdmap e33: 0 total, 0 up, 0 in 2024-01-13T11:55:03.059 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: cephadm 2024-01-13T11:55:01.090136+0000 mgr.smithi088.xwlxdh (mgr.14182) 266 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:55:03.059 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:55:03.059 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:55:03.059 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: return f(*arg) 2024-01-13T11:55:03.059 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:55:03.059 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:55:03.059 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:55:03.059 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: code, '\n'.join(err))) 2024-01-13T11:55:03.059 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:55:03.059 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpm4b2cbzw:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmptrxwfmqp:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:55:03.059 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:55:03.059 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:55:03.059 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:55:03.060 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 86c693ba-57c1-49eb-8ef0-0f2ecc375b2b 2024-01-13T11:55:03.060 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:55:03.060 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:55:03.060 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:55:03.060 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:55:03.060 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:55:03.060 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:55:03.061 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:55:03.061 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:55:03.062 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:55:03.062 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:55:03.062 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:55:03.062 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:55:03.062 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:55:03.062 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:55:03.062 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:55:03.063 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:55:03.063 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:55:03.063 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:55:03.063 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:55:03.063 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:55:03.063 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:55:03.063 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:55:03.063 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:55:03.063 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:55:03.063 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:55:03.064 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:55:03.064 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:55:03.064 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:55:03.064 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:55:03.064 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:55:03.064 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:55:03.064 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:55:03.064 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:55:03.064 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:55:03.064 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:55:03.065 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:55:03.065 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:55:03.065 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:55:03.065 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:55:03.065 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:55:03.065 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:55:03.065 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:55:03.065 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:55:03.065 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:55:03.065 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:55:03.065 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:55:03.065 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:55:03.065 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:55:03.066 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:55:03.066 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:55:03.066 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:55:03.066 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:55:03.066 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:55:03.066 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:55:03.066 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:55:03.066 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:55:03.066 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:55:03.066 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:55:03.066 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: main() 2024-01-13T11:55:03.066 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:55:03.066 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:55:03.067 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:55:03.067 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:55:03.067 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:55:03.067 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:55:03.067 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:55:03.067 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:55:03.067 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:55:03.067 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:55:03.067 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:55:03.067 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:55:03.067 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:55:03.067 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:55:03.067 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpm4b2cbzw:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmptrxwfmqp:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:55:03.068 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: cluster 2024-01-13T11:55:01.091879+0000 mgr.smithi088.xwlxdh (mgr.14182) 267 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:03.068 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: audit 2024-01-13T11:55:01.811041+0000 mon.smithi088 (mon.0) 495 : audit [INF] from='client.? 172.21.15.88:0/3963089486' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17643570-e67e-407b-a20a-0fd649501123"}]: dispatch 2024-01-13T11:55:03.068 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: audit 2024-01-13T11:55:01.816814+0000 mon.smithi088 (mon.0) 496 : audit [INF] from='client.? 172.21.15.88:0/3963089486' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "17643570-e67e-407b-a20a-0fd649501123"}]': finished 2024-01-13T11:55:03.068 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: cluster 2024-01-13T11:55:01.816897+0000 mon.smithi088 (mon.0) 497 : cluster [DBG] osdmap e32: 1 total, 0 up, 1 in 2024-01-13T11:55:03.068 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: audit 2024-01-13T11:55:01.817187+0000 mon.smithi088 (mon.0) 498 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:55:03.068 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: audit 2024-01-13T11:55:02.170176+0000 mon.smithi088 (mon.0) 499 : audit [INF] from='client.? 172.21.15.88:0/2075864320' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:55:03.068 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: audit 2024-01-13T11:55:02.175419+0000 mon.smithi088 (mon.0) 500 : audit [INF] from='client.? 172.21.15.88:0/2075864320' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:55:03.068 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:02 smithi088 bash[14069]: cluster 2024-01-13T11:55:02.175484+0000 mon.smithi088 (mon.0) 501 : cluster [DBG] osdmap e33: 0 total, 0 up, 0 in 2024-01-13T11:55:04.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:03 smithi088 bash[14069]: cluster 2024-01-13T11:55:03.941220+0000 mon.smithi088 (mon.0) 502 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T11:55:04.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:04 smithi138 bash[17620]: cluster 2024-01-13T11:55:03.941220+0000 mon.smithi088 (mon.0) 502 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T11:55:04.805 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:55:05.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: cluster 2024-01-13T11:55:03.092233+0000 mgr.smithi088.xwlxdh (mgr.14182) 268 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:05.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: cephadm 2024-01-13T11:55:03.890480+0000 mgr.smithi088.xwlxdh (mgr.14182) 269 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:55:05.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:55:05.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:55:05.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: return f(*arg) 2024-01-13T11:55:05.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:55:05.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:55:05.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:55:05.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: code, '\n'.join(err))) 2024-01-13T11:55:05.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi088/config 2024-01-13T11:55:05.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpuhviqe50:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpx2bm3c_5:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:55:05.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:55:05.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:55:05.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:55:05.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 17643570-e67e-407b-a20a-0fd649501123 2024-01-13T11:55:05.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:55:05.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:55:05.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:55:05.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:55:05.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:55:05.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:55:05.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:55:05.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:55:05.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:55:05.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:55:05.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:55:05.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:55:05.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:55:05.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:55:05.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:55:05.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:55:05.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:55:05.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:55:05.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:55:05.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:55:05.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:55:05.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:55:05.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:55:05.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:55:05.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:55:05.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:55:05.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:55:05.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:55:05.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:55:05.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:55:05.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:55:05.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:55:05.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:55:05.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:55:05.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:55:05.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:55:05.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:55:05.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:55:05.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:55:05.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:55:05.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:55:05.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:55:05.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:55:05.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:55:05.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:55:05.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:55:05.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:55:05.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:55:05.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:55:05.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:55:05.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:55:05.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:55:05.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:55:05.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:55:05.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:55:05.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:55:05.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:55:05.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:55:05.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: main() 2024-01-13T11:55:05.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:55:05.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:55:05.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:55:05.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:55:05.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:55:05.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:55:05.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:55:05.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:55:05.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:55:05.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:55:05.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:55:05.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:55:05.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:55:05.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:55:05.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpuhviqe50:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpx2bm3c_5:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:55:05.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: cephadm 2024-01-13T11:55:03.893008+0000 mgr.smithi088.xwlxdh (mgr.14182) 270 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T11:55:05.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: service_id: all-available-devices 2024-01-13T11:55:05.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: service_name: osd.all-available-devices 2024-01-13T11:55:05.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: placement: 2024-01-13T11:55:05.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: host_pattern: '*' 2024-01-13T11:55:05.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: spec: 2024-01-13T11:55:05.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: data_devices: 2024-01-13T11:55:05.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: all: true 2024-01-13T11:55:05.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: filter_logic: AND 2024-01-13T11:55:05.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: objectstore: bluestore 2024-01-13T11:55:05.152 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:55:05.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpm4b2cbzw:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmptrxwfmqp:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:55:05.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:55:05.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:55:05.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:55:05.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 86c693ba-57c1-49eb-8ef0-0f2ecc375b2b 2024-01-13T11:55:05.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:55:05.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:55:05.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:55:05.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:55:05.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:55:05.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:55:05.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:55:05.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:55:05.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:55:05.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:55:05.153 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:55:05.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:55:05.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:55:05.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:55:05.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:55:05.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:55:05.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:55:05.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:55:05.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:55:05.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:55:05.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:55:05.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:55:05.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:55:05.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:55:05.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:55:05.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:55:05.154 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:55:05.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:55:05.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:55:05.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:55:05.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:55:05.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:55:05.155 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:55:05.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:55:05.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:55:05.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:55:05.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:55:05.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:55:05.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:55:05.156 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:55:05.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:55:05.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:55:05.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:55:05.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:55:05.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:55:05.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:55:05.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:55:05.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:55:05.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:55:05.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:55:05.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:55:05.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:55:05.157 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:55:05.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:55:05.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:55:05.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:55:05.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:55:05.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:55:05.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: main() 2024-01-13T11:55:05.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:55:05.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:55:05.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:55:05.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:55:05.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:55:05.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:55:05.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:55:05.158 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:55:05.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:55:05.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:55:05.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:55:05.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:55:05.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:55:05.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:55:05.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpm4b2cbzw:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmptrxwfmqp:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:55:05.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:55:05.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T11:55:05.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: if self._apply_service(spec): 2024-01-13T11:55:05.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T11:55:05.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T11:55:05.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T11:55:05.159 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T11:55:05.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T11:55:05.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T11:55:05.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T11:55:05.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T11:55:05.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T11:55:05.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: raise self._value 2024-01-13T11:55:05.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T11:55:05.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: result = (True, func(*args, **kwds)) 2024-01-13T11:55:05.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T11:55:05.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: return list(map(*args)) 2024-01-13T11:55:05.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:55:05.160 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: return f(*arg) 2024-01-13T11:55:05.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:55:05.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:55:05.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:55:05.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: code, '\n'.join(err))) 2024-01-13T11:55:05.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:55:05.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpm4b2cbzw:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmptrxwfmqp:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:55:05.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:55:05.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:55:05.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:55:05.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 86c693ba-57c1-49eb-8ef0-0f2ecc375b2b 2024-01-13T11:55:05.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:55:05.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:55:05.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:55:05.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:55:05.161 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:55:05.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:55:05.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:55:05.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:55:05.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:55:05.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:55:05.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:55:05.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:55:05.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:55:05.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:55:05.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:55:05.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:55:05.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:55:05.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:55:05.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:55:05.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:55:05.162 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:55:05.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:55:05.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:55:05.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:55:05.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:55:05.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:55:05.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:55:05.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:55:05.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:55:05.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:55:05.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:55:05.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:55:05.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:55:05.163 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:55:05.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:55:05.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:55:05.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:55:05.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:55:05.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:55:05.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:55:05.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:55:05.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:55:05.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:55:05.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:55:05.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:55:05.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:55:05.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:55:05.164 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:55:05.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:55:05.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:55:05.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:55:05.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:55:05.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:55:05.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:55:05.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:55:05.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:55:05.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:55:05.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:55:05.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: main() 2024-01-13T11:55:05.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:55:05.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:55:05.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:55:05.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:55:05.165 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:55:05.166 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:55:05.166 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:55:05.166 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:55:05.166 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:55:05.166 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:55:05.166 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:55:05.166 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:55:05.166 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:55:05.166 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:55:05.166 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpm4b2cbzw:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmptrxwfmqp:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:55:05.166 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: cluster 2024-01-13T11:55:03.894483+0000 mgr.smithi088.xwlxdh (mgr.14182) 271 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:05.166 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:05 smithi088 bash[14069]: audit 2024-01-13T11:55:04.817043+0000 mon.smithi088 (mon.0) 503 : audit [DBG] from='client.? 172.21.15.88:0/840373741' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:55:05.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: cluster 2024-01-13T11:55:03.092233+0000 mgr.smithi088.xwlxdh (mgr.14182) 268 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:05.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: cephadm 2024-01-13T11:55:03.890480+0000 mgr.smithi088.xwlxdh (mgr.14182) 269 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:55:05.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:55:05.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:55:05.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: return f(*arg) 2024-01-13T11:55:05.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:55:05.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:55:05.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:55:05.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: code, '\n'.join(err))) 2024-01-13T11:55:05.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi088/config 2024-01-13T11:55:05.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpuhviqe50:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpx2bm3c_5:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:55:05.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:55:05.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:55:05.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:55:05.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 17643570-e67e-407b-a20a-0fd649501123 2024-01-13T11:55:05.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:55:05.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:55:05.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:55:05.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:55:05.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:55:05.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:55:05.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:55:05.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:55:05.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:55:05.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:55:05.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:55:05.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:55:05.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:55:05.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:55:05.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:55:05.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:55:05.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:55:05.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:55:05.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:55:05.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:55:05.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:55:05.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:55:05.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:55:05.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:55:05.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:55:05.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:55:05.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:55:05.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:55:05.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:55:05.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:55:05.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:55:05.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:55:05.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:55:05.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:55:05.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:55:05.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:55:05.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:55:05.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:55:05.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:55:05.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:55:05.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:55:05.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:55:05.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:55:05.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:55:05.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:55:05.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:55:05.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:55:05.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:55:05.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:55:05.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:55:05.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:55:05.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:55:05.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:55:05.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:55:05.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:55:05.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:55:05.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:55:05.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:55:05.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: main() 2024-01-13T11:55:05.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:55:05.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:55:05.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:55:05.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:55:05.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:55:05.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:55:05.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:55:05.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:55:05.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:55:05.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:55:05.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:55:05.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:55:05.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:55:05.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:55:05.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpuhviqe50:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpx2bm3c_5:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:55:05.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: cephadm 2024-01-13T11:55:03.893008+0000 mgr.smithi088.xwlxdh (mgr.14182) 270 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T11:55:05.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: service_id: all-available-devices 2024-01-13T11:55:05.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: service_name: osd.all-available-devices 2024-01-13T11:55:05.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: placement: 2024-01-13T11:55:05.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: host_pattern: '*' 2024-01-13T11:55:05.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: spec: 2024-01-13T11:55:05.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: data_devices: 2024-01-13T11:55:05.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: all: true 2024-01-13T11:55:05.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: filter_logic: AND 2024-01-13T11:55:05.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: objectstore: bluestore 2024-01-13T11:55:05.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:55:05.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpm4b2cbzw:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmptrxwfmqp:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:55:05.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:55:05.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:55:05.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:55:05.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 86c693ba-57c1-49eb-8ef0-0f2ecc375b2b 2024-01-13T11:55:05.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:55:05.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:55:05.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:55:05.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:55:05.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:55:05.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:55:05.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:55:05.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:55:05.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:55:05.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:55:05.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:55:05.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:55:05.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:55:05.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:55:05.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:55:05.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:55:05.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:55:05.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:55:05.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:55:05.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:55:05.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:55:05.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:55:05.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:55:05.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:55:05.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:55:05.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:55:05.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:55:05.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:55:05.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:55:05.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:55:05.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:55:05.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:55:05.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:55:05.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:55:05.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:55:05.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:55:05.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:55:05.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:55:05.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:55:05.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:55:05.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:55:05.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:55:05.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:55:05.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:55:05.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:55:05.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:55:05.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:55:05.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:55:05.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:55:05.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:55:05.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:55:05.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:55:05.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:55:05.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:55:05.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:55:05.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:55:05.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:55:05.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:55:05.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: main() 2024-01-13T11:55:05.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:55:05.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:55:05.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:55:05.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:55:05.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:55:05.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:55:05.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:55:05.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:55:05.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:55:05.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:55:05.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:55:05.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:55:05.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:55:05.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:55:05.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpm4b2cbzw:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmptrxwfmqp:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:55:05.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:55:05.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T11:55:05.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: if self._apply_service(spec): 2024-01-13T11:55:05.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T11:55:05.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T11:55:05.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T11:55:05.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T11:55:05.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T11:55:05.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T11:55:05.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T11:55:05.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T11:55:05.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T11:55:05.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: raise self._value 2024-01-13T11:55:05.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T11:55:05.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: result = (True, func(*args, **kwds)) 2024-01-13T11:55:05.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T11:55:05.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: return list(map(*args)) 2024-01-13T11:55:05.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:55:05.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: return f(*arg) 2024-01-13T11:55:05.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:55:05.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:55:05.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:55:05.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: code, '\n'.join(err))) 2024-01-13T11:55:05.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:55:05.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpm4b2cbzw:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmptrxwfmqp:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:55:05.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:55:05.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:55:05.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:55:05.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 86c693ba-57c1-49eb-8ef0-0f2ecc375b2b 2024-01-13T11:55:05.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:55:05.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:55:05.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:55:05.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:55:05.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:55:05.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:55:05.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:55:05.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:55:05.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:55:05.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:55:05.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:55:05.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:55:05.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:55:05.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:55:05.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:55:05.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:55:05.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:55:05.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:55:05.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:55:05.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:55:05.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:55:05.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:55:05.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:55:05.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:55:05.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:55:05.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:55:05.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:55:05.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:55:05.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:55:05.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:55:05.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:55:05.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:55:05.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:55:05.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:55:05.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:55:05.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:55:05.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:55:05.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:55:05.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:55:05.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:55:05.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:55:05.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:55:05.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:55:05.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:55:05.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:55:05.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:55:05.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:55:05.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:55:05.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:55:05.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:55:05.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:55:05.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:55:05.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:55:05.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:55:05.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:55:05.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:55:05.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:55:05.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:55:05.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: main() 2024-01-13T11:55:05.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:55:05.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:55:05.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:55:05.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:55:05.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:55:05.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:55:05.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:55:05.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:55:05.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:55:05.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:55:05.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:55:05.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:55:05.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:55:05.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:55:05.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpm4b2cbzw:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmptrxwfmqp:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:55:05.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: cluster 2024-01-13T11:55:03.894483+0000 mgr.smithi088.xwlxdh (mgr.14182) 271 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:05.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:05 smithi138 bash[17620]: audit 2024-01-13T11:55:04.817043+0000 mon.smithi088 (mon.0) 503 : audit [DBG] from='client.? 172.21.15.88:0/840373741' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:55:05.602 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:55:06.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:06 smithi138 bash[17620]: cluster 2024-01-13T11:55:05.895482+0000 mgr.smithi088.xwlxdh (mgr.14182) 272 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:06.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:06 smithi088 bash[14069]: cluster 2024-01-13T11:55:05.895482+0000 mgr.smithi088.xwlxdh (mgr.14182) 272 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:06.604 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:55:09.342 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:08 smithi088 bash[14069]: cluster 2024-01-13T11:55:07.896542+0000 mgr.smithi088.xwlxdh (mgr.14182) 273 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:09.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:08 smithi138 bash[17620]: cluster 2024-01-13T11:55:07.896542+0000 mgr.smithi088.xwlxdh (mgr.14182) 273 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:09.696 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:55:10.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:09 smithi088 bash[14069]: audit 2024-01-13T11:55:09.707505+0000 mon.smithi088 (mon.0) 504 : audit [DBG] from='client.? 172.21.15.88:0/2572282547' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:55:10.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:09 smithi138 bash[17620]: audit 2024-01-13T11:55:09.707505+0000 mon.smithi088 (mon.0) 504 : audit [DBG] from='client.? 172.21.15.88:0/2572282547' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:55:10.570 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:55:11.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:10 smithi138 bash[17620]: cluster 2024-01-13T11:55:09.897156+0000 mgr.smithi088.xwlxdh (mgr.14182) 274 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:11.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:10 smithi088 bash[14069]: cluster 2024-01-13T11:55:09.897156+0000 mgr.smithi088.xwlxdh (mgr.14182) 274 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:11.571 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:55:13.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:12 smithi138 bash[17620]: cluster 2024-01-13T11:55:11.898235+0000 mgr.smithi088.xwlxdh (mgr.14182) 275 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:13.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:12 smithi088 bash[14069]: cluster 2024-01-13T11:55:11.898235+0000 mgr.smithi088.xwlxdh (mgr.14182) 275 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:14.736 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:55:15.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:14 smithi088 bash[14069]: cluster 2024-01-13T11:55:13.899241+0000 mgr.smithi088.xwlxdh (mgr.14182) 276 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:15.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:14 smithi088 bash[14069]: audit 2024-01-13T11:55:14.735723+0000 mon.smithi088 (mon.0) 505 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:55:15.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:14 smithi088 bash[14069]: audit 2024-01-13T11:55:14.747712+0000 mon.smithi088 (mon.0) 506 : audit [DBG] from='client.? 172.21.15.88:0/2964338094' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:55:15.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:14 smithi138 bash[17620]: cluster 2024-01-13T11:55:13.899241+0000 mgr.smithi088.xwlxdh (mgr.14182) 276 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:15.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:14 smithi138 bash[17620]: audit 2024-01-13T11:55:14.735723+0000 mon.smithi088 (mon.0) 505 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:55:15.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:14 smithi138 bash[17620]: audit 2024-01-13T11:55:14.747712+0000 mon.smithi088 (mon.0) 506 : audit [DBG] from='client.? 172.21.15.88:0/2964338094' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:55:15.506 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:55:16.507 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:55:17.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:17 smithi138 bash[17620]: cluster 2024-01-13T11:55:15.900185+0000 mgr.smithi088.xwlxdh (mgr.14182) 277 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:17.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:16 smithi088 bash[14069]: cluster 2024-01-13T11:55:15.900185+0000 mgr.smithi088.xwlxdh (mgr.14182) 277 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:19.229 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:18 smithi088 bash[14069]: cluster 2024-01-13T11:55:17.900774+0000 mgr.smithi088.xwlxdh (mgr.14182) 278 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:19.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:18 smithi138 bash[17620]: cluster 2024-01-13T11:55:17.900774+0000 mgr.smithi088.xwlxdh (mgr.14182) 278 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:19.565 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:55:20.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:19 smithi088 bash[14069]: audit 2024-01-13T11:55:19.576421+0000 mon.smithi088 (mon.0) 507 : audit [DBG] from='client.? 172.21.15.88:0/2673299080' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:55:20.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:19 smithi138 bash[17620]: audit 2024-01-13T11:55:19.576421+0000 mon.smithi088 (mon.0) 507 : audit [DBG] from='client.? 172.21.15.88:0/2673299080' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:55:20.374 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:55:21.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:20 smithi138 bash[17620]: cluster 2024-01-13T11:55:19.901469+0000 mgr.smithi088.xwlxdh (mgr.14182) 279 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:21.374 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:55:21.387 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:20 smithi088 bash[14069]: cluster 2024-01-13T11:55:19.901469+0000 mgr.smithi088.xwlxdh (mgr.14182) 279 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:23.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:22 smithi088 bash[14069]: cluster 2024-01-13T11:55:21.902340+0000 mgr.smithi088.xwlxdh (mgr.14182) 280 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:23.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:22 smithi138 bash[17620]: cluster 2024-01-13T11:55:21.902340+0000 mgr.smithi088.xwlxdh (mgr.14182) 280 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:24.608 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:55:25.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:24 smithi138 bash[17620]: cluster 2024-01-13T11:55:23.903272+0000 mgr.smithi088.xwlxdh (mgr.14182) 281 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:25.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:24 smithi138 bash[17620]: audit 2024-01-13T11:55:24.620095+0000 mon.smithi088 (mon.0) 508 : audit [DBG] from='client.? 172.21.15.88:0/2818689155' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:55:25.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:24 smithi088 bash[14069]: cluster 2024-01-13T11:55:23.903272+0000 mgr.smithi088.xwlxdh (mgr.14182) 281 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:25.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:24 smithi088 bash[14069]: audit 2024-01-13T11:55:24.620095+0000 mon.smithi088 (mon.0) 508 : audit [DBG] from='client.? 172.21.15.88:0/2818689155' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:55:25.450 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:55:26.451 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:55:27.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:27 smithi138 bash[17620]: cluster 2024-01-13T11:55:25.904301+0000 mgr.smithi088.xwlxdh (mgr.14182) 282 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:27.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:26 smithi088 bash[14069]: cluster 2024-01-13T11:55:25.904301+0000 mgr.smithi088.xwlxdh (mgr.14182) 282 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:29.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:29 smithi138 bash[17620]: cluster 2024-01-13T11:55:27.904925+0000 mgr.smithi088.xwlxdh (mgr.14182) 283 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:29.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:29 smithi088 bash[14069]: cluster 2024-01-13T11:55:27.904925+0000 mgr.smithi088.xwlxdh (mgr.14182) 283 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:29.827 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:55:30.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:30 smithi088 bash[14069]: audit 2024-01-13T11:55:29.838131+0000 mon.smithi088 (mon.0) 509 : audit [DBG] from='client.? 172.21.15.88:0/2394819737' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:55:30.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:30 smithi088 bash[14069]: cluster 2024-01-13T11:55:29.905863+0000 mgr.smithi088.xwlxdh (mgr.14182) 284 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:30.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:30 smithi138 bash[17620]: audit 2024-01-13T11:55:29.838131+0000 mon.smithi088 (mon.0) 509 : audit [DBG] from='client.? 172.21.15.88:0/2394819737' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:55:30.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:30 smithi138 bash[17620]: cluster 2024-01-13T11:55:29.905863+0000 mgr.smithi088.xwlxdh (mgr.14182) 284 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:30.660 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:55:31.662 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:55:33.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:32 smithi088 bash[14069]: cluster 2024-01-13T11:55:31.906840+0000 mgr.smithi088.xwlxdh (mgr.14182) 285 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:33.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:32 smithi138 bash[17620]: cluster 2024-01-13T11:55:31.906840+0000 mgr.smithi088.xwlxdh (mgr.14182) 285 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:35.073 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:55:35.334 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:34 smithi088 bash[14069]: cluster 2024-01-13T11:55:33.907844+0000 mgr.smithi088.xwlxdh (mgr.14182) 286 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:35.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:34 smithi138 bash[17620]: cluster 2024-01-13T11:55:33.907844+0000 mgr.smithi088.xwlxdh (mgr.14182) 286 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:35.892 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:55:36.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:35 smithi138 bash[17620]: audit 2024-01-13T11:55:35.084791+0000 mon.smithi088 (mon.0) 510 : audit [DBG] from='client.? 172.21.15.88:0/2515496444' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:55:36.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:35 smithi088 bash[14069]: audit 2024-01-13T11:55:35.084791+0000 mon.smithi088 (mon.0) 510 : audit [DBG] from='client.? 172.21.15.88:0/2515496444' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:55:36.895 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:55:37.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:36 smithi138 bash[17620]: cluster 2024-01-13T11:55:35.908797+0000 mgr.smithi088.xwlxdh (mgr.14182) 287 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:37.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:36 smithi088 bash[14069]: cluster 2024-01-13T11:55:35.908797+0000 mgr.smithi088.xwlxdh (mgr.14182) 287 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:39.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:39 smithi138 bash[17620]: cluster 2024-01-13T11:55:37.909613+0000 mgr.smithi088.xwlxdh (mgr.14182) 288 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:39.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:39 smithi088 bash[14069]: cluster 2024-01-13T11:55:37.909613+0000 mgr.smithi088.xwlxdh (mgr.14182) 288 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:40.128 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:55:40.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:40 smithi138 bash[17620]: cluster 2024-01-13T11:55:39.910194+0000 mgr.smithi088.xwlxdh (mgr.14182) 289 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:40.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:40 smithi088 bash[14069]: cluster 2024-01-13T11:55:39.910194+0000 mgr.smithi088.xwlxdh (mgr.14182) 289 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:40.933 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:55:41.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:41 smithi138 bash[17620]: audit 2024-01-13T11:55:40.139660+0000 mon.smithi088 (mon.0) 511 : audit [DBG] from='client.? 172.21.15.88:0/2579744581' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:55:41.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:41 smithi088 bash[14069]: audit 2024-01-13T11:55:40.139660+0000 mon.smithi088 (mon.0) 511 : audit [DBG] from='client.? 172.21.15.88:0/2579744581' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:55:41.934 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:55:42.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:42 smithi138 bash[17620]: cluster 2024-01-13T11:55:41.911165+0000 mgr.smithi088.xwlxdh (mgr.14182) 290 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:42.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:42 smithi088 bash[14069]: cluster 2024-01-13T11:55:41.911165+0000 mgr.smithi088.xwlxdh (mgr.14182) 290 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:45.070 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:55:45.299 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:44 smithi088 bash[14069]: cluster 2024-01-13T11:55:43.912209+0000 mgr.smithi088.xwlxdh (mgr.14182) 291 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:45.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:44 smithi138 bash[17620]: cluster 2024-01-13T11:55:43.912209+0000 mgr.smithi088.xwlxdh (mgr.14182) 291 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:45.852 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:55:46.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:45 smithi088 bash[14069]: audit 2024-01-13T11:55:45.081568+0000 mon.smithi088 (mon.0) 512 : audit [DBG] from='client.? 172.21.15.88:0/3777543846' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:55:46.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:45 smithi138 bash[17620]: audit 2024-01-13T11:55:45.081568+0000 mon.smithi088 (mon.0) 512 : audit [DBG] from='client.? 172.21.15.88:0/3777543846' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:55:46.853 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:55:47.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:46 smithi088 bash[14069]: cluster 2024-01-13T11:55:45.913274+0000 mgr.smithi088.xwlxdh (mgr.14182) 292 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:47.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:46 smithi138 bash[17620]: cluster 2024-01-13T11:55:45.913274+0000 mgr.smithi088.xwlxdh (mgr.14182) 292 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:49.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:49 smithi088 bash[14069]: cluster 2024-01-13T11:55:47.913882+0000 mgr.smithi088.xwlxdh (mgr.14182) 293 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:49.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:49 smithi138 bash[17620]: cluster 2024-01-13T11:55:47.913882+0000 mgr.smithi088.xwlxdh (mgr.14182) 293 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:50.108 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:55:50.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:50 smithi138 bash[17620]: cluster 2024-01-13T11:55:49.914823+0000 mgr.smithi088.xwlxdh (mgr.14182) 294 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:50.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:50 smithi088 bash[14069]: cluster 2024-01-13T11:55:49.914823+0000 mgr.smithi088.xwlxdh (mgr.14182) 294 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:50.963 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:55:51.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:51 smithi138 bash[17620]: audit 2024-01-13T11:55:50.119617+0000 mon.smithi088 (mon.0) 513 : audit [DBG] from='client.? 172.21.15.88:0/3418127223' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:55:51.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:51 smithi088 bash[14069]: audit 2024-01-13T11:55:50.119617+0000 mon.smithi088 (mon.0) 513 : audit [DBG] from='client.? 172.21.15.88:0/3418127223' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:55:51.964 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:55:52.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:52 smithi138 bash[17620]: cluster 2024-01-13T11:55:51.915833+0000 mgr.smithi088.xwlxdh (mgr.14182) 295 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:52.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:52 smithi088 bash[14069]: cluster 2024-01-13T11:55:51.915833+0000 mgr.smithi088.xwlxdh (mgr.14182) 295 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:55.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:54 smithi138 bash[17620]: cluster 2024-01-13T11:55:53.916854+0000 mgr.smithi088.xwlxdh (mgr.14182) 296 : cluster [DBG] pgmap v266: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:55.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:54 smithi088 bash[14069]: cluster 2024-01-13T11:55:53.916854+0000 mgr.smithi088.xwlxdh (mgr.14182) 296 : cluster [DBG] pgmap v266: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:55.397 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:55:56.191 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:55:56.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:55 smithi138 bash[17620]: audit 2024-01-13T11:55:55.409250+0000 mon.smithi088 (mon.0) 514 : audit [DBG] from='client.? 172.21.15.88:0/234862836' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:55:56.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:55 smithi088 bash[14069]: audit 2024-01-13T11:55:55.409250+0000 mon.smithi088 (mon.0) 514 : audit [DBG] from='client.? 172.21.15.88:0/234862836' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:55:57.192 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:55:57.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:56 smithi138 bash[17620]: cluster 2024-01-13T11:55:55.917806+0000 mgr.smithi088.xwlxdh (mgr.14182) 297 : cluster [DBG] pgmap v267: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:57.385 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:56 smithi088 bash[14069]: cluster 2024-01-13T11:55:55.917806+0000 mgr.smithi088.xwlxdh (mgr.14182) 297 : cluster [DBG] pgmap v267: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:59.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:55:59 smithi138 bash[17620]: cluster 2024-01-13T11:55:57.918308+0000 mgr.smithi088.xwlxdh (mgr.14182) 298 : cluster [DBG] pgmap v268: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:55:59.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:55:59 smithi088 bash[14069]: cluster 2024-01-13T11:55:57.918308+0000 mgr.smithi088.xwlxdh (mgr.14182) 298 : cluster [DBG] pgmap v268: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:00.274 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:56:01.046 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:56:01.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:01 smithi138 bash[17620]: cluster 2024-01-13T11:55:59.919376+0000 mgr.smithi088.xwlxdh (mgr.14182) 299 : cluster [DBG] pgmap v269: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:01.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:01 smithi138 bash[17620]: audit 2024-01-13T11:56:00.285575+0000 mon.smithi088 (mon.0) 515 : audit [DBG] from='client.? 172.21.15.88:0/968776295' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:56:01.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:01 smithi088 bash[14069]: cluster 2024-01-13T11:55:59.919376+0000 mgr.smithi088.xwlxdh (mgr.14182) 299 : cluster [DBG] pgmap v269: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:01.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:01 smithi088 bash[14069]: audit 2024-01-13T11:56:00.285575+0000 mon.smithi088 (mon.0) 515 : audit [DBG] from='client.? 172.21.15.88:0/968776295' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:56:02.047 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:56:02.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:02 smithi138 bash[17620]: cluster 2024-01-13T11:56:01.920412+0000 mgr.smithi088.xwlxdh (mgr.14182) 300 : cluster [DBG] pgmap v270: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:02.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:02 smithi088 bash[14069]: cluster 2024-01-13T11:56:01.920412+0000 mgr.smithi088.xwlxdh (mgr.14182) 300 : cluster [DBG] pgmap v270: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:04.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:03 smithi138 bash[17620]: audit 2024-01-13T11:56:03.896738+0000 mon.smithi088 (mon.0) 516 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:56:04.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:03 smithi088 bash[14069]: audit 2024-01-13T11:56:03.896738+0000 mon.smithi088 (mon.0) 516 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:56:05.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:04 smithi138 bash[17620]: cluster 2024-01-13T11:56:03.921350+0000 mgr.smithi088.xwlxdh (mgr.14182) 301 : cluster [DBG] pgmap v271: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:05.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:04 smithi138 bash[17620]: audit 2024-01-13T11:56:04.196894+0000 mon.smithi088 (mon.0) 517 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:56:05.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:04 smithi138 bash[17620]: audit 2024-01-13T11:56:04.198071+0000 mon.smithi088 (mon.0) 518 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T11:56:05.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:04 smithi138 bash[17620]: audit 2024-01-13T11:56:04.205490+0000 mon.smithi088 (mon.0) 519 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:56:05.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:04 smithi138 bash[17620]: audit 2024-01-13T11:56:04.212441+0000 mon.smithi088 (mon.0) 520 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T11:56:05.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:04 smithi138 bash[17620]: audit 2024-01-13T11:56:04.216735+0000 mon.smithi088 (mon.0) 521 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:56:05.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:04 smithi138 bash[17620]: audit 2024-01-13T11:56:04.218312+0000 mon.smithi088 (mon.0) 522 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:56:05.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:04 smithi138 bash[17620]: audit 2024-01-13T11:56:04.218501+0000 mon.smithi088 (mon.0) 523 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:56:05.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:04 smithi138 bash[17620]: audit 2024-01-13T11:56:04.219188+0000 mon.smithi088 (mon.0) 524 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:56:05.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:04 smithi088 bash[14069]: cluster 2024-01-13T11:56:03.921350+0000 mgr.smithi088.xwlxdh (mgr.14182) 301 : cluster [DBG] pgmap v271: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:05.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:04 smithi088 bash[14069]: audit 2024-01-13T11:56:04.196894+0000 mon.smithi088 (mon.0) 517 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:56:05.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:04 smithi088 bash[14069]: audit 2024-01-13T11:56:04.198071+0000 mon.smithi088 (mon.0) 518 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T11:56:05.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:04 smithi088 bash[14069]: audit 2024-01-13T11:56:04.205490+0000 mon.smithi088 (mon.0) 519 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:56:05.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:04 smithi088 bash[14069]: audit 2024-01-13T11:56:04.212441+0000 mon.smithi088 (mon.0) 520 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T11:56:05.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:04 smithi088 bash[14069]: audit 2024-01-13T11:56:04.216735+0000 mon.smithi088 (mon.0) 521 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:56:05.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:04 smithi088 bash[14069]: audit 2024-01-13T11:56:04.218312+0000 mon.smithi088 (mon.0) 522 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:56:05.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:04 smithi088 bash[14069]: audit 2024-01-13T11:56:04.218501+0000 mon.smithi088 (mon.0) 523 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:56:05.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:04 smithi088 bash[14069]: audit 2024-01-13T11:56:04.219188+0000 mon.smithi088 (mon.0) 524 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:56:05.438 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:56:06.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:05 smithi088 bash[14069]: cluster 2024-01-13T11:56:04.206005+0000 mgr.smithi088.xwlxdh (mgr.14182) 302 : cluster [DBG] pgmap v272: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:06.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:05 smithi088 bash[14069]: cluster 2024-01-13T11:56:05.202012+0000 mon.smithi088 (mon.0) 525 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T11:56:06.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:05 smithi088 bash[14069]: cluster 2024-01-13T11:56:05.202075+0000 mon.smithi088 (mon.0) 526 : cluster [INF] Cluster is now healthy 2024-01-13T11:56:06.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:05 smithi088 bash[14069]: audit 2024-01-13T11:56:05.450128+0000 mon.smithi088 (mon.0) 527 : audit [DBG] from='client.? 172.21.15.88:0/2172194223' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:56:06.257 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:56:06.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:05 smithi138 bash[17620]: cluster 2024-01-13T11:56:04.206005+0000 mgr.smithi088.xwlxdh (mgr.14182) 302 : cluster [DBG] pgmap v272: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:06.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:05 smithi138 bash[17620]: cluster 2024-01-13T11:56:05.202012+0000 mon.smithi088 (mon.0) 525 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T11:56:06.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:05 smithi138 bash[17620]: cluster 2024-01-13T11:56:05.202075+0000 mon.smithi088 (mon.0) 526 : cluster [INF] Cluster is now healthy 2024-01-13T11:56:06.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:05 smithi138 bash[17620]: audit 2024-01-13T11:56:05.450128+0000 mon.smithi088 (mon.0) 527 : audit [DBG] from='client.? 172.21.15.88:0/2172194223' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:56:07.258 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:56:08.320 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:08 smithi088 bash[14069]: cluster 2024-01-13T11:56:06.206341+0000 mgr.smithi088.xwlxdh (mgr.14182) 303 : cluster [DBG] pgmap v273: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:08.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:08 smithi138 bash[17620]: cluster 2024-01-13T11:56:06.206341+0000 mgr.smithi088.xwlxdh (mgr.14182) 303 : cluster [DBG] pgmap v273: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:10.273 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:10 smithi138 bash[17620]: cluster 2024-01-13T11:56:08.206758+0000 mgr.smithi088.xwlxdh (mgr.14182) 304 : cluster [DBG] pgmap v274: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:10.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:10 smithi088 bash[14069]: cluster 2024-01-13T11:56:08.206758+0000 mgr.smithi088.xwlxdh (mgr.14182) 304 : cluster [DBG] pgmap v274: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:11.120 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:11 smithi088 bash[14069]: cluster 2024-01-13T11:56:10.207159+0000 mgr.smithi088.xwlxdh (mgr.14182) 305 : cluster [DBG] pgmap v275: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:11.122 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:11 smithi138 bash[17620]: cluster 2024-01-13T11:56:10.207159+0000 mgr.smithi088.xwlxdh (mgr.14182) 305 : cluster [DBG] pgmap v275: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:11.259 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:56:12.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:12 smithi138 bash[17620]: audit 2024-01-13T11:56:11.271383+0000 mon.smithi088 (mon.0) 528 : audit [DBG] from='client.? 172.21.15.88:0/1634660526' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:56:12.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:12 smithi088 bash[14069]: audit 2024-01-13T11:56:11.271383+0000 mon.smithi088 (mon.0) 528 : audit [DBG] from='client.? 172.21.15.88:0/1634660526' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:56:12.556 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:56:13.558 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:56:14.123 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:13 smithi138 bash[17620]: cluster 2024-01-13T11:56:12.207600+0000 mgr.smithi088.xwlxdh (mgr.14182) 306 : cluster [DBG] pgmap v276: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:14.123 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:13 smithi138 bash[17620]: audit 2024-01-13T11:56:13.206644+0000 mon.smithi138 (mon.1) 15 : audit [INF] from='client.? 172.21.15.138:0/2498200470' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52c40a55-7e62-48a0-9929-45a21179c14d"}]: dispatch 2024-01-13T11:56:14.123 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:13 smithi138 bash[17620]: audit 2024-01-13T11:56:13.206876+0000 mon.smithi088 (mon.0) 529 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52c40a55-7e62-48a0-9929-45a21179c14d"}]: dispatch 2024-01-13T11:56:14.123 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:13 smithi138 bash[17620]: audit 2024-01-13T11:56:13.213736+0000 mon.smithi088 (mon.0) 530 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "52c40a55-7e62-48a0-9929-45a21179c14d"}]': finished 2024-01-13T11:56:14.123 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:13 smithi138 bash[17620]: cluster 2024-01-13T11:56:13.213829+0000 mon.smithi088 (mon.0) 531 : cluster [DBG] osdmap e34: 1 total, 0 up, 1 in 2024-01-13T11:56:14.123 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:13 smithi138 bash[17620]: audit 2024-01-13T11:56:13.213982+0000 mon.smithi088 (mon.0) 532 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:56:14.123 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:13 smithi138 bash[17620]: audit 2024-01-13T11:56:13.561056+0000 mon.smithi138 (mon.1) 16 : audit [INF] from='client.? 172.21.15.138:0/3555720285' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:56:14.123 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:13 smithi138 bash[17620]: audit 2024-01-13T11:56:13.561529+0000 mon.smithi088 (mon.0) 533 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:56:14.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:13 smithi138 bash[17620]: audit 2024-01-13T11:56:13.570791+0000 mon.smithi088 (mon.0) 534 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:56:14.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:13 smithi138 bash[17620]: cluster 2024-01-13T11:56:13.570968+0000 mon.smithi088 (mon.0) 535 : cluster [DBG] osdmap e35: 0 total, 0 up, 0 in 2024-01-13T11:56:14.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:13 smithi088 bash[14069]: cluster 2024-01-13T11:56:12.207600+0000 mgr.smithi088.xwlxdh (mgr.14182) 306 : cluster [DBG] pgmap v276: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:14.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:13 smithi088 bash[14069]: audit 2024-01-13T11:56:13.206644+0000 mon.smithi138 (mon.1) 15 : audit [INF] from='client.? 172.21.15.138:0/2498200470' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52c40a55-7e62-48a0-9929-45a21179c14d"}]: dispatch 2024-01-13T11:56:14.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:13 smithi088 bash[14069]: audit 2024-01-13T11:56:13.206876+0000 mon.smithi088 (mon.0) 529 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52c40a55-7e62-48a0-9929-45a21179c14d"}]: dispatch 2024-01-13T11:56:14.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:13 smithi088 bash[14069]: audit 2024-01-13T11:56:13.213736+0000 mon.smithi088 (mon.0) 530 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "52c40a55-7e62-48a0-9929-45a21179c14d"}]': finished 2024-01-13T11:56:14.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:13 smithi088 bash[14069]: cluster 2024-01-13T11:56:13.213829+0000 mon.smithi088 (mon.0) 531 : cluster [DBG] osdmap e34: 1 total, 0 up, 1 in 2024-01-13T11:56:14.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:13 smithi088 bash[14069]: audit 2024-01-13T11:56:13.213982+0000 mon.smithi088 (mon.0) 532 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:56:14.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:13 smithi088 bash[14069]: audit 2024-01-13T11:56:13.561056+0000 mon.smithi138 (mon.1) 16 : audit [INF] from='client.? 172.21.15.138:0/3555720285' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:56:14.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:13 smithi088 bash[14069]: audit 2024-01-13T11:56:13.561529+0000 mon.smithi088 (mon.0) 533 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:56:14.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:13 smithi088 bash[14069]: audit 2024-01-13T11:56:13.570791+0000 mon.smithi088 (mon.0) 534 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:56:14.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:13 smithi088 bash[14069]: cluster 2024-01-13T11:56:13.570968+0000 mon.smithi088 (mon.0) 535 : cluster [DBG] osdmap e35: 0 total, 0 up, 0 in 2024-01-13T11:56:15.119 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:14 smithi088 bash[14069]: audit 2024-01-13T11:56:14.575021+0000 mon.smithi088 (mon.0) 536 : audit [INF] from='client.? 172.21.15.88:0/360601982' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40a58721-ca0b-4466-b95d-5605ee9d1865"}]: dispatch 2024-01-13T11:56:15.119 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:14 smithi088 bash[14069]: audit 2024-01-13T11:56:14.582756+0000 mon.smithi088 (mon.0) 537 : audit [INF] from='client.? 172.21.15.88:0/360601982' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "40a58721-ca0b-4466-b95d-5605ee9d1865"}]': finished 2024-01-13T11:56:15.119 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:14 smithi088 bash[14069]: cluster 2024-01-13T11:56:14.582808+0000 mon.smithi088 (mon.0) 538 : cluster [DBG] osdmap e36: 1 total, 0 up, 1 in 2024-01-13T11:56:15.119 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:14 smithi088 bash[14069]: audit 2024-01-13T11:56:14.582899+0000 mon.smithi088 (mon.0) 539 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:56:15.120 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:14 smithi088 bash[14069]: audit 2024-01-13T11:56:14.735434+0000 mon.smithi088 (mon.0) 540 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:56:15.120 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:14 smithi088 bash[14069]: audit 2024-01-13T11:56:14.930296+0000 mon.smithi088 (mon.0) 541 : audit [INF] from='client.? 172.21.15.88:0/4161229455' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:56:15.120 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:14 smithi088 bash[14069]: audit 2024-01-13T11:56:14.940188+0000 mon.smithi088 (mon.0) 542 : audit [INF] from='client.? 172.21.15.88:0/4161229455' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:56:15.120 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:14 smithi088 bash[14069]: cluster 2024-01-13T11:56:14.940273+0000 mon.smithi088 (mon.0) 543 : cluster [DBG] osdmap e37: 0 total, 0 up, 0 in 2024-01-13T11:56:15.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:14 smithi138 bash[17620]: audit 2024-01-13T11:56:14.575021+0000 mon.smithi088 (mon.0) 536 : audit [INF] from='client.? 172.21.15.88:0/360601982' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40a58721-ca0b-4466-b95d-5605ee9d1865"}]: dispatch 2024-01-13T11:56:15.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:14 smithi138 bash[17620]: audit 2024-01-13T11:56:14.582756+0000 mon.smithi088 (mon.0) 537 : audit [INF] from='client.? 172.21.15.88:0/360601982' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "40a58721-ca0b-4466-b95d-5605ee9d1865"}]': finished 2024-01-13T11:56:15.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:14 smithi138 bash[17620]: cluster 2024-01-13T11:56:14.582808+0000 mon.smithi088 (mon.0) 538 : cluster [DBG] osdmap e36: 1 total, 0 up, 1 in 2024-01-13T11:56:15.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:14 smithi138 bash[17620]: audit 2024-01-13T11:56:14.582899+0000 mon.smithi088 (mon.0) 539 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:56:15.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:14 smithi138 bash[17620]: audit 2024-01-13T11:56:14.735434+0000 mon.smithi088 (mon.0) 540 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:56:15.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:14 smithi138 bash[17620]: audit 2024-01-13T11:56:14.930296+0000 mon.smithi088 (mon.0) 541 : audit [INF] from='client.? 172.21.15.88:0/4161229455' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:56:15.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:14 smithi138 bash[17620]: audit 2024-01-13T11:56:14.940188+0000 mon.smithi088 (mon.0) 542 : audit [INF] from='client.? 172.21.15.88:0/4161229455' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:56:15.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:14 smithi138 bash[17620]: cluster 2024-01-13T11:56:14.940273+0000 mon.smithi088 (mon.0) 543 : cluster [DBG] osdmap e37: 0 total, 0 up, 0 in 2024-01-13T11:56:16.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:15 smithi088 bash[14069]: cluster 2024-01-13T11:56:14.207973+0000 mgr.smithi088.xwlxdh (mgr.14182) 307 : cluster [DBG] pgmap v279: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:16.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:15 smithi088 bash[14069]: cephadm 2024-01-13T11:56:14.482283+0000 mgr.smithi088.xwlxdh (mgr.14182) 308 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:56:16.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:15 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:56:16.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:15 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:56:16.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:15 smithi088 bash[14069]: return f(*arg) 2024-01-13T11:56:16.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:15 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:56:16.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:15 smithi088 bash[14069]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:56:16.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:15 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:56:16.141 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:15 smithi088 bash[14069]: code, '\n'.join(err))) 2024-01-13T11:56:16.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:15 smithi088 bash[14069]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:56:16.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:15 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpnvbneq82:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpmjw5172u:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:56:16.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:15 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:56:16.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:15 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:56:16.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:15 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:56:16.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:15 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 52c40a55-7e62-48a0-9929-45a21179c14d 2024-01-13T11:56:16.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:15 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:56:16.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:15 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:56:16.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:15 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:56:16.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:15 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:56:16.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:15 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:56:16.142 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:56:16.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:56:16.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:56:16.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:56:16.143 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:56:16.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:56:16.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:56:16.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:56:16.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:56:16.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:56:16.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:56:16.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:56:16.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:56:16.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:56:16.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:56:16.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:56:16.144 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:56:16.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:56:16.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:56:16.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:56:16.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:56:16.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:56:16.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:56:16.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:56:16.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:56:16.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:56:16.145 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:56:16.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:56:16.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:56:16.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:56:16.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:56:16.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:56:16.146 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:56:16.147 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:56:16.147 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:56:16.147 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:56:16.147 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:56:16.147 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:56:16.147 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:56:16.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:56:16.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:56:16.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:56:16.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:56:16.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:56:16.148 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:56:16.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:56:16.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:56:16.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:56:16.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:56:16.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:56:16.149 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:56:16.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:56:16.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:56:16.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: main() 2024-01-13T11:56:16.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:56:16.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:56:16.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:56:16.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:56:16.150 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:56:16.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:56:16.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:56:16.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:56:16.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:56:16.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:56:16.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:56:16.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:56:16.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:56:16.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:56:16.151 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:16 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpnvbneq82:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpmjw5172u:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:56:16.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: cluster 2024-01-13T11:56:14.207973+0000 mgr.smithi088.xwlxdh (mgr.14182) 307 : cluster [DBG] pgmap v279: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:16.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: cephadm 2024-01-13T11:56:14.482283+0000 mgr.smithi088.xwlxdh (mgr.14182) 308 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:56:16.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:56:16.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:56:16.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: return f(*arg) 2024-01-13T11:56:16.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:56:16.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:56:16.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:56:16.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: code, '\n'.join(err))) 2024-01-13T11:56:16.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:56:16.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpnvbneq82:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpmjw5172u:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:56:16.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:56:16.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:56:16.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:56:16.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 52c40a55-7e62-48a0-9929-45a21179c14d 2024-01-13T11:56:16.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:56:16.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:56:16.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:56:16.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:56:16.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:56:16.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:56:16.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:56:16.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:56:16.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:56:16.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:56:16.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:56:16.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:56:16.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:56:16.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:56:16.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:56:16.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:56:16.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:56:16.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:56:16.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:56:16.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:56:16.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:56:16.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:56:16.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:56:16.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:56:16.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:56:16.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:56:16.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:56:16.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:56:16.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:56:16.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:56:16.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:56:16.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:56:16.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:56:16.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:56:16.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:56:16.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:56:16.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:56:16.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:56:16.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:56:16.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:56:16.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:56:16.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:56:16.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:56:16.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:56:16.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:56:16.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:56:16.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:56:16.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:56:16.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:56:16.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:56:16.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:56:16.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:56:16.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:56:16.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:56:16.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:56:16.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:56:16.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:56:16.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:56:16.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: main() 2024-01-13T11:56:16.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:56:16.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:56:16.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:56:16.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:56:16.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:56:16.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:56:16.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:56:16.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:56:16.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:56:16.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:56:16.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:56:16.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:56:16.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:56:16.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:56:16.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:16 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpnvbneq82:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpmjw5172u:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:56:17.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: cluster 2024-01-13T11:56:16.208414+0000 mgr.smithi088.xwlxdh (mgr.14182) 309 : cluster [DBG] pgmap v282: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:17.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: cephadm 2024-01-13T11:56:16.449365+0000 mgr.smithi088.xwlxdh (mgr.14182) 310 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:56:17.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:56:17.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:56:17.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: return f(*arg) 2024-01-13T11:56:17.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:56:17.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:56:17.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:56:17.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: code, '\n'.join(err))) 2024-01-13T11:56:17.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi088/config 2024-01-13T11:56:17.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpom3v7ffe:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp8c1sw783:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:56:17.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:56:17.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:56:17.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:56:17.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 40a58721-ca0b-4466-b95d-5605ee9d1865 2024-01-13T11:56:17.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:56:17.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:56:17.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:56:17.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:56:17.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:56:17.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:56:17.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:56:17.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:56:17.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:56:17.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:56:17.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:56:17.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:56:17.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:56:17.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:56:17.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:56:17.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:56:17.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:56:17.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:56:17.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:56:17.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:56:17.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:56:17.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:56:17.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:56:17.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:56:17.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:56:17.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:56:17.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:56:17.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:56:17.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:56:17.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:56:17.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:56:17.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:56:17.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:56:17.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:56:17.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:56:17.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:56:17.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:56:17.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:56:17.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:56:17.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:56:17.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:56:17.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:56:17.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:56:17.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:56:17.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:56:17.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:56:17.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:56:17.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:56:17.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:56:17.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:56:17.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:56:17.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:56:17.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:56:17.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:56:17.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:56:17.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:56:17.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:56:17.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:56:17.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: main() 2024-01-13T11:56:17.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:56:17.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:56:17.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:56:17.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:56:17.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:56:17.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:56:17.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:56:17.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:56:17.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:56:17.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:56:17.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:56:17.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:56:17.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:56:17.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:56:17.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpom3v7ffe:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp8c1sw783:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:56:17.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: cephadm 2024-01-13T11:56:16.451484+0000 mgr.smithi088.xwlxdh (mgr.14182) 311 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T11:56:17.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: service_id: all-available-devices 2024-01-13T11:56:17.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: service_name: osd.all-available-devices 2024-01-13T11:56:17.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: placement: 2024-01-13T11:56:17.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: host_pattern: '*' 2024-01-13T11:56:17.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: spec: 2024-01-13T11:56:17.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: data_devices: 2024-01-13T11:56:17.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: all: true 2024-01-13T11:56:17.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: filter_logic: AND 2024-01-13T11:56:17.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: objectstore: bluestore 2024-01-13T11:56:17.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:56:17.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpnvbneq82:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpmjw5172u:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:56:17.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:56:17.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:56:17.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:56:17.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 52c40a55-7e62-48a0-9929-45a21179c14d 2024-01-13T11:56:17.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:56:17.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:56:17.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:56:17.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:56:17.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:56:17.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:56:17.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:56:17.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:56:17.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:56:17.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:56:17.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:56:17.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:56:17.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:56:17.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:56:17.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:56:17.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:56:17.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:56:17.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:56:17.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:56:17.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:56:17.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:56:17.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:56:17.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:56:17.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:56:17.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:56:17.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:56:17.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:56:17.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:56:17.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:56:17.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:56:17.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:56:17.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:56:17.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:56:17.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:56:17.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:56:17.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:56:17.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:56:17.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:56:17.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:56:17.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:56:17.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:56:17.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:56:17.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:56:17.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:56:17.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:56:17.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:56:17.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:56:17.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:56:17.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:56:17.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:56:17.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:56:17.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:56:17.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:56:17.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:56:17.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:56:17.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:56:17.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:56:17.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:56:17.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: main() 2024-01-13T11:56:17.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:56:17.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:56:17.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:56:17.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:56:17.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:56:17.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:56:17.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:56:17.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:56:17.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:56:17.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:56:17.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:56:17.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:56:17.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:56:17.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:56:17.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpnvbneq82:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpmjw5172u:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:56:17.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:56:17.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T11:56:17.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: if self._apply_service(spec): 2024-01-13T11:56:17.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T11:56:17.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T11:56:17.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T11:56:17.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T11:56:17.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T11:56:17.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T11:56:17.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T11:56:17.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T11:56:17.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T11:56:17.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: raise self._value 2024-01-13T11:56:17.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T11:56:17.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: result = (True, func(*args, **kwds)) 2024-01-13T11:56:17.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T11:56:17.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: return list(map(*args)) 2024-01-13T11:56:17.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:56:17.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: return f(*arg) 2024-01-13T11:56:17.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:56:17.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:56:17.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:56:17.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: code, '\n'.join(err))) 2024-01-13T11:56:17.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:56:17.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpnvbneq82:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpmjw5172u:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:56:17.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:56:17.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:56:17.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:56:17.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 52c40a55-7e62-48a0-9929-45a21179c14d 2024-01-13T11:56:17.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:56:17.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:56:17.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:56:17.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:56:17.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:56:17.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:56:17.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:56:17.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:56:17.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:56:17.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:56:17.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:56:17.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:56:17.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:56:17.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:56:17.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:56:17.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:56:17.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:56:17.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:56:17.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:56:17.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:56:17.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:56:17.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:56:17.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:56:17.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:56:17.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:56:17.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:56:17.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:56:17.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:56:17.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:56:17.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:56:17.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:56:17.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:56:17.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:56:17.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:56:17.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:56:17.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:56:17.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:56:17.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:56:17.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:56:17.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:56:17.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:56:17.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:56:17.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:56:17.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:56:17.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:56:17.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:56:17.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:56:17.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:56:17.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:56:17.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:56:17.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:56:17.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:56:17.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:56:17.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:56:17.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:56:17.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:56:17.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:56:17.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:56:17.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: main() 2024-01-13T11:56:17.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:56:17.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:56:17.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:56:17.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:56:17.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:56:17.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:56:17.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:56:17.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:56:17.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:56:17.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:56:17.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:56:17.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:56:17.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:56:17.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:56:17.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpnvbneq82:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpmjw5172u:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:56:17.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: cluster 2024-01-13T11:56:16.452253+0000 mgr.smithi088.xwlxdh (mgr.14182) 312 : cluster [DBG] pgmap v283: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:17.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:17 smithi138 bash[17620]: cluster 2024-01-13T11:56:16.997748+0000 mon.smithi088 (mon.0) 544 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T11:56:17.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: cluster 2024-01-13T11:56:16.208414+0000 mgr.smithi088.xwlxdh (mgr.14182) 309 : cluster [DBG] pgmap v282: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:17.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: cephadm 2024-01-13T11:56:16.449365+0000 mgr.smithi088.xwlxdh (mgr.14182) 310 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:56:17.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:56:17.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:56:17.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: return f(*arg) 2024-01-13T11:56:17.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:56:17.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:56:17.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:56:17.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: code, '\n'.join(err))) 2024-01-13T11:56:17.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi088/config 2024-01-13T11:56:17.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpom3v7ffe:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp8c1sw783:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:56:17.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:56:17.399 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:56:17.399 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:56:17.399 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 40a58721-ca0b-4466-b95d-5605ee9d1865 2024-01-13T11:56:17.399 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:56:17.399 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:56:17.399 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:56:17.399 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:56:17.399 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:56:17.399 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:56:17.399 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:56:17.399 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:56:17.399 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:56:17.399 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:56:17.400 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:56:17.400 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:56:17.400 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:56:17.400 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:56:17.400 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:56:17.400 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:56:17.400 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:56:17.400 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:56:17.400 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:56:17.400 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:56:17.400 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:56:17.400 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:56:17.400 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:56:17.401 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:56:17.401 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:56:17.401 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:56:17.401 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:56:17.401 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:56:17.401 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:56:17.401 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:56:17.401 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:56:17.401 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:56:17.401 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:56:17.401 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:56:17.401 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:56:17.402 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:56:17.402 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:56:17.402 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:56:17.402 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:56:17.402 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:56:17.402 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:56:17.402 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:56:17.402 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:56:17.402 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:56:17.402 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:56:17.402 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:56:17.402 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:56:17.402 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:56:17.403 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:56:17.403 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:56:17.403 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:56:17.403 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:56:17.403 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:56:17.403 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:56:17.403 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:56:17.403 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:56:17.403 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:56:17.403 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:56:17.403 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: main() 2024-01-13T11:56:17.403 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:56:17.403 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:56:17.403 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:56:17.404 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:56:17.404 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:56:17.404 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:56:17.404 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:56:17.404 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:56:17.404 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:56:17.404 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:56:17.404 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:56:17.404 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:56:17.404 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:56:17.404 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:56:17.404 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpom3v7ffe:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp8c1sw783:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:56:17.404 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: cephadm 2024-01-13T11:56:16.451484+0000 mgr.smithi088.xwlxdh (mgr.14182) 311 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T11:56:17.404 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: service_id: all-available-devices 2024-01-13T11:56:17.405 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: service_name: osd.all-available-devices 2024-01-13T11:56:17.405 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: placement: 2024-01-13T11:56:17.405 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: host_pattern: '*' 2024-01-13T11:56:17.405 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: spec: 2024-01-13T11:56:17.405 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: data_devices: 2024-01-13T11:56:17.405 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: all: true 2024-01-13T11:56:17.405 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: filter_logic: AND 2024-01-13T11:56:17.405 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: objectstore: bluestore 2024-01-13T11:56:17.405 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:56:17.405 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpnvbneq82:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpmjw5172u:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:56:17.405 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:56:17.405 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:56:17.405 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:56:17.405 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 52c40a55-7e62-48a0-9929-45a21179c14d 2024-01-13T11:56:17.406 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:56:17.406 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:56:17.406 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:56:17.406 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:56:17.406 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:56:17.406 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:56:17.406 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:56:17.406 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:56:17.406 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:56:17.406 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:56:17.406 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:56:17.406 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:56:17.406 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:56:17.406 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:56:17.406 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:56:17.407 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:56:17.407 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:56:17.407 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:56:17.407 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:56:17.407 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:56:17.407 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:56:17.407 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:56:17.407 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:56:17.407 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:56:17.407 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:56:17.407 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:56:17.408 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:56:17.408 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:56:17.408 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:56:17.408 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:56:17.408 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:56:17.408 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:56:17.408 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:56:17.408 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:56:17.408 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:56:17.408 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:56:17.408 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:56:17.408 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:56:17.408 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:56:17.409 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:56:17.409 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:56:17.409 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:56:17.409 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:56:17.409 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:56:17.409 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:56:17.409 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:56:17.409 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:56:17.409 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:56:17.409 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:56:17.409 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:56:17.409 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:56:17.409 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:56:17.409 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:56:17.410 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:56:17.410 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:56:17.410 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:56:17.410 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:56:17.410 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:56:17.410 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: main() 2024-01-13T11:56:17.410 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:56:17.410 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:56:17.410 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:56:17.410 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:56:17.410 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:56:17.410 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:56:17.410 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:56:17.410 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:56:17.411 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:56:17.411 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:56:17.411 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:56:17.411 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:56:17.411 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:56:17.411 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:56:17.411 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpnvbneq82:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpmjw5172u:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:56:17.411 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:56:17.411 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T11:56:17.411 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: if self._apply_service(spec): 2024-01-13T11:56:17.411 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T11:56:17.411 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T11:56:17.412 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T11:56:17.412 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T11:56:17.412 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T11:56:17.412 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T11:56:17.412 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T11:56:17.412 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T11:56:17.412 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T11:56:17.412 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: raise self._value 2024-01-13T11:56:17.412 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T11:56:17.412 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: result = (True, func(*args, **kwds)) 2024-01-13T11:56:17.412 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T11:56:17.412 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: return list(map(*args)) 2024-01-13T11:56:17.412 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:56:17.413 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: return f(*arg) 2024-01-13T11:56:17.413 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:56:17.413 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:56:17.413 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:56:17.413 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: code, '\n'.join(err))) 2024-01-13T11:56:17.413 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:56:17.413 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpnvbneq82:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpmjw5172u:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:56:17.413 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:56:17.413 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:56:17.413 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:56:17.413 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 52c40a55-7e62-48a0-9929-45a21179c14d 2024-01-13T11:56:17.413 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:56:17.413 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:56:17.414 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:56:17.414 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:56:17.414 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:56:17.414 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:56:17.414 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:56:17.414 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:56:17.414 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:56:17.414 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:56:17.414 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:56:17.414 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:56:17.414 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:56:17.414 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:56:17.414 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:56:17.415 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:56:17.415 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:56:17.415 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:56:17.415 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:56:17.415 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:56:17.415 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:56:17.415 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:56:17.415 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:56:17.415 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:56:17.415 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:56:17.415 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:56:17.415 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:56:17.415 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:56:17.415 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:56:17.416 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:56:17.416 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:56:17.416 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:56:17.416 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:56:17.416 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:56:17.416 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:56:17.416 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:56:17.416 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:56:17.416 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:56:17.416 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:56:17.416 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:56:17.416 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:56:17.417 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:56:17.417 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:56:17.417 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:56:17.417 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:56:17.417 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:56:17.417 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:56:17.417 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:56:17.417 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:56:17.417 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:56:17.417 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:56:17.417 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:56:17.417 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:56:17.417 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:56:17.417 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:56:17.418 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:56:17.418 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:56:17.418 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:56:17.418 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: main() 2024-01-13T11:56:17.418 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:56:17.418 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:56:17.418 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:56:17.418 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:56:17.418 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:56:17.418 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:56:17.418 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:56:17.418 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:56:17.418 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:56:17.418 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:56:17.419 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:56:17.419 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:56:17.419 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:56:17.419 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:56:17.419 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpnvbneq82:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpmjw5172u:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:56:17.419 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: cluster 2024-01-13T11:56:16.452253+0000 mgr.smithi088.xwlxdh (mgr.14182) 312 : cluster [DBG] pgmap v283: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:17.419 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:17 smithi088 bash[14069]: cluster 2024-01-13T11:56:16.997748+0000 mon.smithi088 (mon.0) 544 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T11:56:17.789 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:56:18.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:18 smithi088 bash[14069]: audit 2024-01-13T11:56:17.801466+0000 mon.smithi088 (mon.0) 545 : audit [DBG] from='client.? 172.21.15.88:0/1704668691' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:56:18.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:18 smithi138 bash[17620]: audit 2024-01-13T11:56:17.801466+0000 mon.smithi088 (mon.0) 545 : audit [DBG] from='client.? 172.21.15.88:0/1704668691' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:56:18.618 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":37,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:56:19.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:19 smithi138 bash[17620]: cluster 2024-01-13T11:56:18.453212+0000 mgr.smithi088.xwlxdh (mgr.14182) 313 : cluster [DBG] pgmap v284: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:19.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:19 smithi088 bash[14069]: cluster 2024-01-13T11:56:18.453212+0000 mgr.smithi088.xwlxdh (mgr.14182) 313 : cluster [DBG] pgmap v284: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:19.619 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:56:21.872 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:21 smithi138 bash[17620]: cluster 2024-01-13T11:56:20.454238+0000 mgr.smithi088.xwlxdh (mgr.14182) 314 : cluster [DBG] pgmap v285: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:21.890 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:21 smithi088 bash[14069]: cluster 2024-01-13T11:56:20.454238+0000 mgr.smithi088.xwlxdh (mgr.14182) 314 : cluster [DBG] pgmap v285: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:22.816 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:56:23.605 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":37,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:56:23.872 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:23 smithi138 bash[17620]: cluster 2024-01-13T11:56:22.454868+0000 mgr.smithi088.xwlxdh (mgr.14182) 315 : cluster [DBG] pgmap v286: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:23.873 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:23 smithi138 bash[17620]: audit 2024-01-13T11:56:22.828177+0000 mon.smithi088 (mon.0) 546 : audit [DBG] from='client.? 172.21.15.88:0/347016016' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:56:23.890 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:23 smithi088 bash[14069]: cluster 2024-01-13T11:56:22.454868+0000 mgr.smithi088.xwlxdh (mgr.14182) 315 : cluster [DBG] pgmap v286: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:23.890 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:23 smithi088 bash[14069]: audit 2024-01-13T11:56:22.828177+0000 mon.smithi088 (mon.0) 546 : audit [DBG] from='client.? 172.21.15.88:0/347016016' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:56:24.607 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:56:25.872 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:25 smithi138 bash[17620]: cluster 2024-01-13T11:56:24.455883+0000 mgr.smithi088.xwlxdh (mgr.14182) 316 : cluster [DBG] pgmap v287: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:25.890 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:25 smithi088 bash[14069]: cluster 2024-01-13T11:56:24.455883+0000 mgr.smithi088.xwlxdh (mgr.14182) 316 : cluster [DBG] pgmap v287: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:27.872 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:27 smithi138 bash[17620]: cluster 2024-01-13T11:56:26.456934+0000 mgr.smithi088.xwlxdh (mgr.14182) 317 : cluster [DBG] pgmap v288: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:27.890 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:27 smithi088 bash[14069]: cluster 2024-01-13T11:56:26.456934+0000 mgr.smithi088.xwlxdh (mgr.14182) 317 : cluster [DBG] pgmap v288: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:27.987 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:56:28.820 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":37,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:56:29.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:28 smithi088 bash[14069]: audit 2024-01-13T11:56:27.999725+0000 mon.smithi088 (mon.0) 547 : audit [DBG] from='client.? 172.21.15.88:0/3273101217' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:56:29.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:28 smithi138 bash[17620]: audit 2024-01-13T11:56:27.999725+0000 mon.smithi088 (mon.0) 547 : audit [DBG] from='client.? 172.21.15.88:0/3273101217' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:56:29.821 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:56:30.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:30 smithi088 bash[14069]: cluster 2024-01-13T11:56:28.457993+0000 mgr.smithi088.xwlxdh (mgr.14182) 318 : cluster [DBG] pgmap v289: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:30.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:30 smithi088 bash[14069]: audit 2024-01-13T11:56:29.737423+0000 mon.smithi088 (mon.0) 548 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:56:30.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:30 smithi138 bash[17620]: cluster 2024-01-13T11:56:28.457993+0000 mgr.smithi088.xwlxdh (mgr.14182) 318 : cluster [DBG] pgmap v289: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:30.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:30 smithi138 bash[17620]: audit 2024-01-13T11:56:29.737423+0000 mon.smithi088 (mon.0) 548 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:56:32.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:32 smithi088 bash[14069]: cluster 2024-01-13T11:56:30.459059+0000 mgr.smithi088.xwlxdh (mgr.14182) 319 : cluster [DBG] pgmap v290: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:32.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:32 smithi138 bash[17620]: cluster 2024-01-13T11:56:30.459059+0000 mgr.smithi088.xwlxdh (mgr.14182) 319 : cluster [DBG] pgmap v290: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:32.926 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:56:33.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:33 smithi138 bash[17620]: audit 2024-01-13T11:56:32.937814+0000 mon.smithi088 (mon.0) 549 : audit [DBG] from='client.? 172.21.15.88:0/4271763287' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:56:33.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:33 smithi088 bash[14069]: audit 2024-01-13T11:56:32.937814+0000 mon.smithi088 (mon.0) 549 : audit [DBG] from='client.? 172.21.15.88:0/4271763287' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:56:33.728 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":37,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:56:34.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:34 smithi138 bash[17620]: cluster 2024-01-13T11:56:32.460096+0000 mgr.smithi088.xwlxdh (mgr.14182) 320 : cluster [DBG] pgmap v291: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:34.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:34 smithi088 bash[14069]: cluster 2024-01-13T11:56:32.460096+0000 mgr.smithi088.xwlxdh (mgr.14182) 320 : cluster [DBG] pgmap v291: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:34.730 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:56:35.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:35 smithi138 bash[17620]: cluster 2024-01-13T11:56:34.461071+0000 mgr.smithi088.xwlxdh (mgr.14182) 321 : cluster [DBG] pgmap v292: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:35.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:35 smithi088 bash[14069]: cluster 2024-01-13T11:56:34.461071+0000 mgr.smithi088.xwlxdh (mgr.14182) 321 : cluster [DBG] pgmap v292: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:37.872 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:37 smithi138 bash[17620]: cluster 2024-01-13T11:56:36.461653+0000 mgr.smithi088.xwlxdh (mgr.14182) 322 : cluster [DBG] pgmap v293: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:37.890 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:37 smithi088 bash[14069]: cluster 2024-01-13T11:56:36.461653+0000 mgr.smithi088.xwlxdh (mgr.14182) 322 : cluster [DBG] pgmap v293: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:38.053 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:56:38.833 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":37,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:56:39.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:38 smithi088 bash[14069]: audit 2024-01-13T11:56:38.065341+0000 mon.smithi088 (mon.0) 550 : audit [DBG] from='client.? 172.21.15.88:0/2194508882' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:56:39.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:38 smithi138 bash[17620]: audit 2024-01-13T11:56:38.065341+0000 mon.smithi088 (mon.0) 550 : audit [DBG] from='client.? 172.21.15.88:0/2194508882' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:56:39.834 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:56:40.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:39 smithi088 bash[14069]: cluster 2024-01-13T11:56:38.462599+0000 mgr.smithi088.xwlxdh (mgr.14182) 323 : cluster [DBG] pgmap v294: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:40.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:40 smithi138 bash[17620]: cluster 2024-01-13T11:56:38.462599+0000 mgr.smithi088.xwlxdh (mgr.14182) 323 : cluster [DBG] pgmap v294: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:42.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:42 smithi088 bash[14069]: cluster 2024-01-13T11:56:40.463652+0000 mgr.smithi088.xwlxdh (mgr.14182) 324 : cluster [DBG] pgmap v295: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:42.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:42 smithi138 bash[17620]: cluster 2024-01-13T11:56:40.463652+0000 mgr.smithi088.xwlxdh (mgr.14182) 324 : cluster [DBG] pgmap v295: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:42.993 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:56:43.782 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":37,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:56:44.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:43 smithi088 bash[14069]: cluster 2024-01-13T11:56:42.464726+0000 mgr.smithi088.xwlxdh (mgr.14182) 325 : cluster [DBG] pgmap v296: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:44.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:43 smithi088 bash[14069]: audit 2024-01-13T11:56:43.005110+0000 mon.smithi088 (mon.0) 551 : audit [DBG] from='client.? 172.21.15.88:0/1364306411' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:56:44.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:43 smithi138 bash[17620]: cluster 2024-01-13T11:56:42.464726+0000 mgr.smithi088.xwlxdh (mgr.14182) 325 : cluster [DBG] pgmap v296: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:44.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:43 smithi138 bash[17620]: audit 2024-01-13T11:56:43.005110+0000 mon.smithi088 (mon.0) 551 : audit [DBG] from='client.? 172.21.15.88:0/1364306411' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:56:44.783 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:56:46.126 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:45 smithi088 bash[14069]: cluster 2024-01-13T11:56:44.465760+0000 mgr.smithi088.xwlxdh (mgr.14182) 326 : cluster [DBG] pgmap v297: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:46.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:46 smithi138 bash[17620]: cluster 2024-01-13T11:56:44.465760+0000 mgr.smithi088.xwlxdh (mgr.14182) 326 : cluster [DBG] pgmap v297: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:47.836 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:56:48.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:48 smithi088 bash[14069]: cluster 2024-01-13T11:56:46.466603+0000 mgr.smithi088.xwlxdh (mgr.14182) 327 : cluster [DBG] pgmap v298: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:48.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:48 smithi088 bash[14069]: audit 2024-01-13T11:56:47.848108+0000 mon.smithi088 (mon.0) 552 : audit [DBG] from='client.? 172.21.15.88:0/3484029303' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:56:48.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:48 smithi138 bash[17620]: cluster 2024-01-13T11:56:46.466603+0000 mgr.smithi088.xwlxdh (mgr.14182) 327 : cluster [DBG] pgmap v298: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:48.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:48 smithi138 bash[17620]: audit 2024-01-13T11:56:47.848108+0000 mon.smithi088 (mon.0) 552 : audit [DBG] from='client.? 172.21.15.88:0/3484029303' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:56:48.615 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":37,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:56:49.617 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:56:50.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:50 smithi088 bash[14069]: cluster 2024-01-13T11:56:48.467656+0000 mgr.smithi088.xwlxdh (mgr.14182) 328 : cluster [DBG] pgmap v299: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:50.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:50 smithi138 bash[17620]: cluster 2024-01-13T11:56:48.467656+0000 mgr.smithi088.xwlxdh (mgr.14182) 328 : cluster [DBG] pgmap v299: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:51.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:51 smithi088 bash[14069]: cluster 2024-01-13T11:56:50.468668+0000 mgr.smithi088.xwlxdh (mgr.14182) 329 : cluster [DBG] pgmap v300: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:51.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:51 smithi138 bash[17620]: cluster 2024-01-13T11:56:50.468668+0000 mgr.smithi088.xwlxdh (mgr.14182) 329 : cluster [DBG] pgmap v300: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:53.045 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:56:53.912 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":37,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:56:54.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:54 smithi138 bash[17620]: cluster 2024-01-13T11:56:52.469716+0000 mgr.smithi088.xwlxdh (mgr.14182) 330 : cluster [DBG] pgmap v301: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:54.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:54 smithi138 bash[17620]: audit 2024-01-13T11:56:53.057280+0000 mon.smithi088 (mon.0) 553 : audit [DBG] from='client.? 172.21.15.88:0/3909125972' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:56:54.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:53 smithi088 bash[14069]: cluster 2024-01-13T11:56:52.469716+0000 mgr.smithi088.xwlxdh (mgr.14182) 330 : cluster [DBG] pgmap v301: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:54.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:53 smithi088 bash[14069]: audit 2024-01-13T11:56:53.057280+0000 mon.smithi088 (mon.0) 553 : audit [DBG] from='client.? 172.21.15.88:0/3909125972' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:56:54.913 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:56:56.243 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:56 smithi088 bash[14069]: cluster 2024-01-13T11:56:54.470819+0000 mgr.smithi088.xwlxdh (mgr.14182) 331 : cluster [DBG] pgmap v302: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:56.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:56 smithi138 bash[17620]: cluster 2024-01-13T11:56:54.470819+0000 mgr.smithi088.xwlxdh (mgr.14182) 331 : cluster [DBG] pgmap v302: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:58.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:58 smithi088 bash[14069]: cluster 2024-01-13T11:56:56.471798+0000 mgr.smithi088.xwlxdh (mgr.14182) 332 : cluster [DBG] pgmap v303: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:58.234 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:56:58.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:58 smithi138 bash[17620]: cluster 2024-01-13T11:56:56.471798+0000 mgr.smithi088.xwlxdh (mgr.14182) 332 : cluster [DBG] pgmap v303: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:56:59.049 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":37,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:56:59.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:56:59 smithi138 bash[17620]: audit 2024-01-13T11:56:58.246213+0000 mon.smithi088 (mon.0) 554 : audit [DBG] from='client.? 172.21.15.88:0/686151174' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:56:59.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:56:59 smithi088 bash[14069]: audit 2024-01-13T11:56:58.246213+0000 mon.smithi088 (mon.0) 554 : audit [DBG] from='client.? 172.21.15.88:0/686151174' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:57:00.050 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:57:00.286 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:00 smithi088 bash[14069]: cluster 2024-01-13T11:56:58.472825+0000 mgr.smithi088.xwlxdh (mgr.14182) 333 : cluster [DBG] pgmap v304: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:00.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:00 smithi138 bash[17620]: cluster 2024-01-13T11:56:58.472825+0000 mgr.smithi088.xwlxdh (mgr.14182) 333 : cluster [DBG] pgmap v304: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:01.336 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:01 smithi088 bash[14069]: cluster 2024-01-13T11:57:00.473861+0000 mgr.smithi088.xwlxdh (mgr.14182) 334 : cluster [DBG] pgmap v305: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:01.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:01 smithi138 bash[17620]: cluster 2024-01-13T11:57:00.473861+0000 mgr.smithi088.xwlxdh (mgr.14182) 334 : cluster [DBG] pgmap v305: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:03.267 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:57:04.104 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":37,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:57:04.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:04 smithi138 bash[17620]: cluster 2024-01-13T11:57:02.474917+0000 mgr.smithi088.xwlxdh (mgr.14182) 335 : cluster [DBG] pgmap v306: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:04.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:04 smithi138 bash[17620]: audit 2024-01-13T11:57:03.278978+0000 mon.smithi088 (mon.0) 555 : audit [DBG] from='client.? 172.21.15.88:0/2502296023' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:57:04.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:03 smithi088 bash[14069]: cluster 2024-01-13T11:57:02.474917+0000 mgr.smithi088.xwlxdh (mgr.14182) 335 : cluster [DBG] pgmap v306: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:04.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:03 smithi088 bash[14069]: audit 2024-01-13T11:57:03.278978+0000 mon.smithi088 (mon.0) 555 : audit [DBG] from='client.? 172.21.15.88:0/2502296023' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:57:05.105 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:57:06.266 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:06 smithi088 bash[14069]: cluster 2024-01-13T11:57:04.475977+0000 mgr.smithi088.xwlxdh (mgr.14182) 336 : cluster [DBG] pgmap v307: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:06.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:06 smithi138 bash[17620]: cluster 2024-01-13T11:57:04.475977+0000 mgr.smithi088.xwlxdh (mgr.14182) 336 : cluster [DBG] pgmap v307: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:08.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:08 smithi088 bash[14069]: cluster 2024-01-13T11:57:06.476624+0000 mgr.smithi088.xwlxdh (mgr.14182) 337 : cluster [DBG] pgmap v308: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:08.199 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:57:08.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:08 smithi138 bash[17620]: cluster 2024-01-13T11:57:06.476624+0000 mgr.smithi088.xwlxdh (mgr.14182) 337 : cluster [DBG] pgmap v308: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:08.998 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":37,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:57:09.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:09 smithi138 bash[17620]: audit 2024-01-13T11:57:08.210872+0000 mon.smithi088 (mon.0) 556 : audit [DBG] from='client.? 172.21.15.88:0/967668092' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:57:09.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:09 smithi088 bash[14069]: audit 2024-01-13T11:57:08.210872+0000 mon.smithi088 (mon.0) 556 : audit [DBG] from='client.? 172.21.15.88:0/967668092' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:57:09.999 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:57:10.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:10 smithi138 bash[17620]: cluster 2024-01-13T11:57:08.477638+0000 mgr.smithi088.xwlxdh (mgr.14182) 338 : cluster [DBG] pgmap v309: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:10.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:10 smithi088 bash[14069]: cluster 2024-01-13T11:57:08.477638+0000 mgr.smithi088.xwlxdh (mgr.14182) 338 : cluster [DBG] pgmap v309: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:11.303 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:11 smithi088 bash[14069]: cluster 2024-01-13T11:57:10.478670+0000 mgr.smithi088.xwlxdh (mgr.14182) 339 : cluster [DBG] pgmap v310: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:11.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:11 smithi138 bash[17620]: cluster 2024-01-13T11:57:10.478670+0000 mgr.smithi088.xwlxdh (mgr.14182) 339 : cluster [DBG] pgmap v310: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:13.337 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:57:14.139 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":37,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:57:14.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:14 smithi138 bash[17620]: cluster 2024-01-13T11:57:12.479896+0000 mgr.smithi088.xwlxdh (mgr.14182) 340 : cluster [DBG] pgmap v311: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:14.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:14 smithi138 bash[17620]: audit 2024-01-13T11:57:13.349725+0000 mon.smithi088 (mon.0) 557 : audit [DBG] from='client.? 172.21.15.88:0/2744875059' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:57:14.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:14 smithi088 bash[14069]: cluster 2024-01-13T11:57:12.479896+0000 mgr.smithi088.xwlxdh (mgr.14182) 340 : cluster [DBG] pgmap v311: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:14.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:14 smithi088 bash[14069]: audit 2024-01-13T11:57:13.349725+0000 mon.smithi088 (mon.0) 557 : audit [DBG] from='client.? 172.21.15.88:0/2744875059' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:57:15.143 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:57:16.266 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:16 smithi088 bash[14069]: cluster 2024-01-13T11:57:14.480936+0000 mgr.smithi088.xwlxdh (mgr.14182) 341 : cluster [DBG] pgmap v312: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:16.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:16 smithi138 bash[17620]: cluster 2024-01-13T11:57:14.480936+0000 mgr.smithi088.xwlxdh (mgr.14182) 341 : cluster [DBG] pgmap v312: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:17.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:17 smithi138 bash[17620]: audit 2024-01-13T11:57:16.454223+0000 mon.smithi088 (mon.0) 558 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:57:17.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:17 smithi138 bash[17620]: audit 2024-01-13T11:57:16.708014+0000 mon.smithi088 (mon.0) 559 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:57:17.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:17 smithi138 bash[17620]: audit 2024-01-13T11:57:16.968793+0000 mon.smithi088 (mon.0) 560 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:57:17.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:17 smithi138 bash[17620]: audit 2024-01-13T11:57:16.970408+0000 mon.smithi088 (mon.0) 561 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T11:57:17.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:17 smithi138 bash[17620]: audit 2024-01-13T11:57:16.978174+0000 mon.smithi088 (mon.0) 562 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:57:17.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:17 smithi138 bash[17620]: audit 2024-01-13T11:57:16.982891+0000 mon.smithi088 (mon.0) 563 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T11:57:17.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:17 smithi138 bash[17620]: audit 2024-01-13T11:57:16.986847+0000 mon.smithi088 (mon.0) 564 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:57:17.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:17 smithi138 bash[17620]: audit 2024-01-13T11:57:16.988955+0000 mon.smithi088 (mon.0) 565 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:57:17.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:17 smithi138 bash[17620]: audit 2024-01-13T11:57:16.989411+0000 mon.smithi088 (mon.0) 566 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:57:17.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:17 smithi138 bash[17620]: audit 2024-01-13T11:57:16.990237+0000 mon.smithi088 (mon.0) 567 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:57:17.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:17 smithi088 bash[14069]: audit 2024-01-13T11:57:16.454223+0000 mon.smithi088 (mon.0) 558 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:57:17.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:17 smithi088 bash[14069]: audit 2024-01-13T11:57:16.708014+0000 mon.smithi088 (mon.0) 559 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:57:17.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:17 smithi088 bash[14069]: audit 2024-01-13T11:57:16.968793+0000 mon.smithi088 (mon.0) 560 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:57:17.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:17 smithi088 bash[14069]: audit 2024-01-13T11:57:16.970408+0000 mon.smithi088 (mon.0) 561 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T11:57:17.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:17 smithi088 bash[14069]: audit 2024-01-13T11:57:16.978174+0000 mon.smithi088 (mon.0) 562 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:57:17.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:17 smithi088 bash[14069]: audit 2024-01-13T11:57:16.982891+0000 mon.smithi088 (mon.0) 563 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T11:57:17.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:17 smithi088 bash[14069]: audit 2024-01-13T11:57:16.986847+0000 mon.smithi088 (mon.0) 564 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:57:17.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:17 smithi088 bash[14069]: audit 2024-01-13T11:57:16.988955+0000 mon.smithi088 (mon.0) 565 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:57:17.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:17 smithi088 bash[14069]: audit 2024-01-13T11:57:16.989411+0000 mon.smithi088 (mon.0) 566 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:57:17.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:17 smithi088 bash[14069]: audit 2024-01-13T11:57:16.990237+0000 mon.smithi088 (mon.0) 567 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:57:18.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:18 smithi138 bash[17620]: cluster 2024-01-13T11:57:16.481991+0000 mgr.smithi088.xwlxdh (mgr.14182) 342 : cluster [DBG] pgmap v313: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:18.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:18 smithi138 bash[17620]: cluster 2024-01-13T11:57:16.978639+0000 mgr.smithi088.xwlxdh (mgr.14182) 343 : cluster [DBG] pgmap v314: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:18.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:18 smithi138 bash[17620]: cluster 2024-01-13T11:57:17.009705+0000 mon.smithi088 (mon.0) 568 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T11:57:18.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:18 smithi138 bash[17620]: cluster 2024-01-13T11:57:17.009790+0000 mon.smithi088 (mon.0) 569 : cluster [INF] Cluster is now healthy 2024-01-13T11:57:18.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:18 smithi088 bash[14069]: cluster 2024-01-13T11:57:16.481991+0000 mgr.smithi088.xwlxdh (mgr.14182) 342 : cluster [DBG] pgmap v313: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:18.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:18 smithi088 bash[14069]: cluster 2024-01-13T11:57:16.978639+0000 mgr.smithi088.xwlxdh (mgr.14182) 343 : cluster [DBG] pgmap v314: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:18.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:18 smithi088 bash[14069]: cluster 2024-01-13T11:57:17.009705+0000 mon.smithi088 (mon.0) 568 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T11:57:18.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:18 smithi088 bash[14069]: cluster 2024-01-13T11:57:17.009790+0000 mon.smithi088 (mon.0) 569 : cluster [INF] Cluster is now healthy 2024-01-13T11:57:18.422 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:57:19.279 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":37,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:57:19.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:19 smithi138 bash[17620]: audit 2024-01-13T11:57:18.433869+0000 mon.smithi088 (mon.0) 570 : audit [DBG] from='client.? 172.21.15.88:0/1597264590' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:57:19.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:19 smithi138 bash[17620]: cluster 2024-01-13T11:57:18.979028+0000 mgr.smithi088.xwlxdh (mgr.14182) 344 : cluster [DBG] pgmap v315: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:19.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:19 smithi088 bash[14069]: audit 2024-01-13T11:57:18.433869+0000 mon.smithi088 (mon.0) 570 : audit [DBG] from='client.? 172.21.15.88:0/1597264590' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:57:19.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:19 smithi088 bash[14069]: cluster 2024-01-13T11:57:18.979028+0000 mgr.smithi088.xwlxdh (mgr.14182) 344 : cluster [DBG] pgmap v315: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:20.280 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:57:21.353 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:21 smithi088 bash[14069]: cluster 2024-01-13T11:57:20.979406+0000 mgr.smithi088.xwlxdh (mgr.14182) 345 : cluster [DBG] pgmap v316: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:21.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:21 smithi138 bash[17620]: cluster 2024-01-13T11:57:20.979406+0000 mgr.smithi088.xwlxdh (mgr.14182) 345 : cluster [DBG] pgmap v316: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:24.122 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:24 smithi138 bash[17620]: cluster 2024-01-13T11:57:22.979850+0000 mgr.smithi088.xwlxdh (mgr.14182) 346 : cluster [DBG] pgmap v317: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:24.255 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:57:24.279 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:24 smithi088 bash[14069]: cluster 2024-01-13T11:57:22.979850+0000 mgr.smithi088.xwlxdh (mgr.14182) 346 : cluster [DBG] pgmap v317: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:25.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:25 smithi138 bash[17620]: audit 2024-01-13T11:57:24.267002+0000 mon.smithi088 (mon.0) 571 : audit [DBG] from='client.? 172.21.15.88:0/1437607351' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:57:25.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:25 smithi088 bash[14069]: audit 2024-01-13T11:57:24.267002+0000 mon.smithi088 (mon.0) 571 : audit [DBG] from='client.? 172.21.15.88:0/1437607351' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:57:25.577 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":37,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:57:26.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:26 smithi138 bash[17620]: cluster 2024-01-13T11:57:24.980351+0000 mgr.smithi088.xwlxdh (mgr.14182) 347 : cluster [DBG] pgmap v318: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:26.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:26 smithi138 bash[17620]: audit 2024-01-13T11:57:25.511386+0000 mon.smithi138 (mon.1) 17 : audit [INF] from='client.? 172.21.15.138:0/1549034404' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0353b9e9-d2bc-4178-83b7-92f6a0d0c6e9"}]: dispatch 2024-01-13T11:57:26.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:26 smithi138 bash[17620]: audit 2024-01-13T11:57:25.511758+0000 mon.smithi088 (mon.0) 572 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0353b9e9-d2bc-4178-83b7-92f6a0d0c6e9"}]: dispatch 2024-01-13T11:57:26.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:26 smithi138 bash[17620]: audit 2024-01-13T11:57:25.519920+0000 mon.smithi088 (mon.0) 573 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0353b9e9-d2bc-4178-83b7-92f6a0d0c6e9"}]': finished 2024-01-13T11:57:26.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:26 smithi138 bash[17620]: cluster 2024-01-13T11:57:25.520062+0000 mon.smithi088 (mon.0) 574 : cluster [DBG] osdmap e38: 1 total, 0 up, 1 in 2024-01-13T11:57:26.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:26 smithi138 bash[17620]: audit 2024-01-13T11:57:25.520288+0000 mon.smithi088 (mon.0) 575 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:57:26.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:26 smithi138 bash[17620]: audit 2024-01-13T11:57:25.874117+0000 mon.smithi138 (mon.1) 18 : audit [INF] from='client.? 172.21.15.138:0/1255398171' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:57:26.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:26 smithi138 bash[17620]: audit 2024-01-13T11:57:25.874356+0000 mon.smithi088 (mon.0) 576 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:57:26.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:26 smithi138 bash[17620]: audit 2024-01-13T11:57:25.882068+0000 mon.smithi088 (mon.0) 577 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:57:26.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:26 smithi138 bash[17620]: cluster 2024-01-13T11:57:25.882150+0000 mon.smithi088 (mon.0) 578 : cluster [DBG] osdmap e39: 0 total, 0 up, 0 in 2024-01-13T11:57:26.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:26 smithi088 bash[14069]: cluster 2024-01-13T11:57:24.980351+0000 mgr.smithi088.xwlxdh (mgr.14182) 347 : cluster [DBG] pgmap v318: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:26.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:26 smithi088 bash[14069]: audit 2024-01-13T11:57:25.511386+0000 mon.smithi138 (mon.1) 17 : audit [INF] from='client.? 172.21.15.138:0/1549034404' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0353b9e9-d2bc-4178-83b7-92f6a0d0c6e9"}]: dispatch 2024-01-13T11:57:26.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:26 smithi088 bash[14069]: audit 2024-01-13T11:57:25.511758+0000 mon.smithi088 (mon.0) 572 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0353b9e9-d2bc-4178-83b7-92f6a0d0c6e9"}]: dispatch 2024-01-13T11:57:26.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:26 smithi088 bash[14069]: audit 2024-01-13T11:57:25.519920+0000 mon.smithi088 (mon.0) 573 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0353b9e9-d2bc-4178-83b7-92f6a0d0c6e9"}]': finished 2024-01-13T11:57:26.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:26 smithi088 bash[14069]: cluster 2024-01-13T11:57:25.520062+0000 mon.smithi088 (mon.0) 574 : cluster [DBG] osdmap e38: 1 total, 0 up, 1 in 2024-01-13T11:57:26.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:26 smithi088 bash[14069]: audit 2024-01-13T11:57:25.520288+0000 mon.smithi088 (mon.0) 575 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:57:26.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:26 smithi088 bash[14069]: audit 2024-01-13T11:57:25.874117+0000 mon.smithi138 (mon.1) 18 : audit [INF] from='client.? 172.21.15.138:0/1255398171' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:57:26.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:26 smithi088 bash[14069]: audit 2024-01-13T11:57:25.874356+0000 mon.smithi088 (mon.0) 576 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:57:26.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:26 smithi088 bash[14069]: audit 2024-01-13T11:57:25.882068+0000 mon.smithi088 (mon.0) 577 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:57:26.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:26 smithi088 bash[14069]: cluster 2024-01-13T11:57:25.882150+0000 mon.smithi088 (mon.0) 578 : cluster [DBG] osdmap e39: 0 total, 0 up, 0 in 2024-01-13T11:57:26.578 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:57:27.021 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: cephadm 2024-01-13T11:57:26.751849+0000 mgr.smithi088.xwlxdh (mgr.14182) 348 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:57:27.021 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:57:27.021 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:57:27.021 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: return f(*arg) 2024-01-13T11:57:27.021 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:57:27.021 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:57:27.022 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:57:27.022 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: code, '\n'.join(err))) 2024-01-13T11:57:27.022 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:57:27.022 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp2luebpqo:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpf9b7x28t:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:57:27.022 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:57:27.022 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:57:27.022 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:57:27.022 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 0353b9e9-d2bc-4178-83b7-92f6a0d0c6e9 2024-01-13T11:57:27.022 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:57:27.022 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:57:27.022 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:57:27.022 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:57:27.023 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:57:27.023 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:57:27.023 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:57:27.023 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:57:27.023 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:57:27.023 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:57:27.023 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:57:27.023 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:57:27.023 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:57:27.023 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:57:27.023 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:57:27.024 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:57:27.024 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:57:27.024 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:57:27.024 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:57:27.024 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:57:27.024 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:57:27.024 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:57:27.024 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:57:27.024 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:57:27.024 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:57:27.024 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:57:27.025 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:57:27.025 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:57:27.025 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:57:27.025 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:57:27.025 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:57:27.025 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:57:27.025 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:57:27.025 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:57:27.025 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:57:27.025 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:57:27.025 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:57:27.026 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:57:27.026 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:57:27.026 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:57:27.026 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:57:27.027 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:57:27.027 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:57:27.027 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:57:27.027 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:57:27.027 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:57:27.027 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:57:27.027 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:57:27.027 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:57:27.027 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:57:27.027 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:57:27.028 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:57:27.028 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:57:27.028 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:57:27.028 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:57:27.028 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:57:27.028 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:57:27.028 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:57:27.028 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: main() 2024-01-13T11:57:27.028 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:57:27.028 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:57:27.028 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:57:27.029 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:57:27.029 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:57:27.029 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:57:27.029 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:57:27.029 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:57:27.029 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:57:27.029 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:57:27.029 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:57:27.029 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:57:27.029 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:57:27.029 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:57:27.029 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp2luebpqo:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpf9b7x28t:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:57:27.029 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:27 smithi088 bash[14069]: cluster 2024-01-13T11:57:26.980789+0000 mgr.smithi088.xwlxdh (mgr.14182) 349 : cluster [DBG] pgmap v321: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:27.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: cephadm 2024-01-13T11:57:26.751849+0000 mgr.smithi088.xwlxdh (mgr.14182) 348 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:57:27.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:57:27.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:57:27.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: return f(*arg) 2024-01-13T11:57:27.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:57:27.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:57:27.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:57:27.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: code, '\n'.join(err))) 2024-01-13T11:57:27.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:57:27.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp2luebpqo:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpf9b7x28t:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:57:27.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:57:27.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:57:27.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:57:27.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 0353b9e9-d2bc-4178-83b7-92f6a0d0c6e9 2024-01-13T11:57:27.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:57:27.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:57:27.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:57:27.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:57:27.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:57:27.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:57:27.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:57:27.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:57:27.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:57:27.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:57:27.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:57:27.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:57:27.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:57:27.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:57:27.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:57:27.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:57:27.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:57:27.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:57:27.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:57:27.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:57:27.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:57:27.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:57:27.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:57:27.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:57:27.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:57:27.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:57:27.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:57:27.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:57:27.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:57:27.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:57:27.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:57:27.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:57:27.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:57:27.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:57:27.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:57:27.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:57:27.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:57:27.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:57:27.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:57:27.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:57:27.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:57:27.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:57:27.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:57:27.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:57:27.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:57:27.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:57:27.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:57:27.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:57:27.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:57:27.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:57:27.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:57:27.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:57:27.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:57:27.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:57:27.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:57:27.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:57:27.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:57:27.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:57:27.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: main() 2024-01-13T11:57:27.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:57:27.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:57:27.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:57:27.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:57:27.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:57:27.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:57:27.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:57:27.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:57:27.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:57:27.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:57:27.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:57:27.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:57:27.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:57:27.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:57:27.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp2luebpqo:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpf9b7x28t:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:57:27.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:27 smithi138 bash[17620]: cluster 2024-01-13T11:57:26.980789+0000 mgr.smithi088.xwlxdh (mgr.14182) 349 : cluster [DBG] pgmap v321: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:28.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:28 smithi138 bash[17620]: audit 2024-01-13T11:57:27.583083+0000 mon.smithi088 (mon.0) 579 : audit [INF] from='client.? 172.21.15.88:0/1440018396' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9461ac01-d08d-44df-a942-4eba5cd6e3a3"}]: dispatch 2024-01-13T11:57:28.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:28 smithi138 bash[17620]: audit 2024-01-13T11:57:27.590113+0000 mon.smithi088 (mon.0) 580 : audit [INF] from='client.? 172.21.15.88:0/1440018396' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9461ac01-d08d-44df-a942-4eba5cd6e3a3"}]': finished 2024-01-13T11:57:28.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:28 smithi138 bash[17620]: cluster 2024-01-13T11:57:27.590248+0000 mon.smithi088 (mon.0) 581 : cluster [DBG] osdmap e40: 1 total, 0 up, 1 in 2024-01-13T11:57:28.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:28 smithi138 bash[17620]: audit 2024-01-13T11:57:27.590417+0000 mon.smithi088 (mon.0) 582 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:57:28.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:28 smithi138 bash[17620]: audit 2024-01-13T11:57:27.939160+0000 mon.smithi088 (mon.0) 583 : audit [INF] from='client.? 172.21.15.88:0/3137806774' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:57:28.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:28 smithi138 bash[17620]: audit 2024-01-13T11:57:27.943828+0000 mon.smithi088 (mon.0) 584 : audit [INF] from='client.? 172.21.15.88:0/3137806774' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:57:28.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:28 smithi138 bash[17620]: cluster 2024-01-13T11:57:27.943890+0000 mon.smithi088 (mon.0) 585 : cluster [DBG] osdmap e41: 0 total, 0 up, 0 in 2024-01-13T11:57:28.374 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:28 smithi088 bash[14069]: audit 2024-01-13T11:57:27.583083+0000 mon.smithi088 (mon.0) 579 : audit [INF] from='client.? 172.21.15.88:0/1440018396' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9461ac01-d08d-44df-a942-4eba5cd6e3a3"}]: dispatch 2024-01-13T11:57:28.374 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:28 smithi088 bash[14069]: audit 2024-01-13T11:57:27.590113+0000 mon.smithi088 (mon.0) 580 : audit [INF] from='client.? 172.21.15.88:0/1440018396' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9461ac01-d08d-44df-a942-4eba5cd6e3a3"}]': finished 2024-01-13T11:57:28.374 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:28 smithi088 bash[14069]: cluster 2024-01-13T11:57:27.590248+0000 mon.smithi088 (mon.0) 581 : cluster [DBG] osdmap e40: 1 total, 0 up, 1 in 2024-01-13T11:57:28.375 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:28 smithi088 bash[14069]: audit 2024-01-13T11:57:27.590417+0000 mon.smithi088 (mon.0) 582 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:57:28.375 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:28 smithi088 bash[14069]: audit 2024-01-13T11:57:27.939160+0000 mon.smithi088 (mon.0) 583 : audit [INF] from='client.? 172.21.15.88:0/3137806774' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:57:28.375 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:28 smithi088 bash[14069]: audit 2024-01-13T11:57:27.943828+0000 mon.smithi088 (mon.0) 584 : audit [INF] from='client.? 172.21.15.88:0/3137806774' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:57:28.375 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:28 smithi088 bash[14069]: cluster 2024-01-13T11:57:27.943890+0000 mon.smithi088 (mon.0) 585 : cluster [DBG] osdmap e41: 0 total, 0 up, 0 in 2024-01-13T11:57:29.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:29 smithi138 bash[17620]: cluster 2024-01-13T11:57:28.981178+0000 mgr.smithi088.xwlxdh (mgr.14182) 350 : cluster [DBG] pgmap v324: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:29.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:29 smithi088 bash[14069]: cluster 2024-01-13T11:57:28.981178+0000 mgr.smithi088.xwlxdh (mgr.14182) 350 : cluster [DBG] pgmap v324: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:30.859 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:57:31.100 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: cephadm 2024-01-13T11:57:29.452238+0000 mgr.smithi088.xwlxdh (mgr.14182) 351 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:57:31.100 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:57:31.100 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:57:31.100 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: return f(*arg) 2024-01-13T11:57:31.100 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:57:31.100 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:57:31.100 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:57:31.101 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: code, '\n'.join(err))) 2024-01-13T11:57:31.101 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi088/config 2024-01-13T11:57:31.101 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpg7qhqtho:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp28z3abgb:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:57:31.101 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:57:31.101 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:57:31.101 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:57:31.101 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 9461ac01-d08d-44df-a942-4eba5cd6e3a3 2024-01-13T11:57:31.101 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:57:31.101 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:57:31.101 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:57:31.101 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:57:31.101 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:57:31.101 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:57:31.101 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:57:31.102 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:57:31.102 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:57:31.102 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:57:31.102 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:57:31.102 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:57:31.102 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:57:31.102 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:57:31.102 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:57:31.102 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:57:31.102 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:57:31.102 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:57:31.102 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:57:31.102 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:57:31.102 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:57:31.103 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:57:31.103 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:57:31.103 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:57:31.103 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:57:31.103 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:57:31.103 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:57:31.103 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:57:31.103 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:57:31.103 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:57:31.103 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:57:31.103 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:57:31.103 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:57:31.103 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:57:31.103 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:57:31.104 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:57:31.104 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:57:31.104 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:57:31.104 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:57:31.104 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:57:31.104 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:57:31.105 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:57:31.105 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:57:31.105 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:57:31.105 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:57:31.106 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:57:31.106 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:57:31.106 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:57:31.106 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:57:31.106 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:57:31.106 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:57:31.106 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:57:31.106 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:57:31.106 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:57:31.106 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:57:31.106 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:57:31.106 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:57:31.106 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:57:31.106 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: main() 2024-01-13T11:57:31.107 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:57:31.107 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:57:31.107 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:57:31.107 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:57:31.107 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:57:31.107 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:57:31.107 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:57:31.107 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:57:31.107 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:57:31.107 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:57:31.107 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:57:31.107 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:57:31.107 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:57:31.107 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:57:31.108 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpg7qhqtho:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp28z3abgb:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:57:31.108 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: cephadm 2024-01-13T11:57:29.455525+0000 mgr.smithi088.xwlxdh (mgr.14182) 352 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T11:57:31.108 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: service_id: all-available-devices 2024-01-13T11:57:31.108 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: service_name: osd.all-available-devices 2024-01-13T11:57:31.108 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: placement: 2024-01-13T11:57:31.108 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: host_pattern: '*' 2024-01-13T11:57:31.108 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: spec: 2024-01-13T11:57:31.108 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: data_devices: 2024-01-13T11:57:31.108 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: all: true 2024-01-13T11:57:31.108 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: filter_logic: AND 2024-01-13T11:57:31.108 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: objectstore: bluestore 2024-01-13T11:57:31.108 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:57:31.108 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp2luebpqo:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpf9b7x28t:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:57:31.109 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:57:31.109 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:57:31.109 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:57:31.109 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 0353b9e9-d2bc-4178-83b7-92f6a0d0c6e9 2024-01-13T11:57:31.109 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:57:31.109 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:57:31.109 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:57:31.109 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:57:31.109 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:57:31.109 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:57:31.109 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:57:31.109 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:57:31.109 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:57:31.109 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:57:31.110 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:57:31.110 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:57:31.110 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:57:31.110 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:57:31.110 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:57:31.110 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:57:31.110 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:57:31.110 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:57:31.110 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:57:31.110 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:57:31.110 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:57:31.110 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:57:31.110 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:57:31.110 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:57:31.110 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:57:31.111 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:57:31.111 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:57:31.111 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:57:31.111 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:57:31.111 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:57:31.111 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:57:31.111 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:57:31.111 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:57:31.111 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:57:31.111 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:57:31.111 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:57:31.111 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:57:31.111 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:57:31.111 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:57:31.112 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:57:31.112 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:57:31.112 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:57:31.112 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:57:31.112 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:57:31.112 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:57:31.112 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:57:31.112 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:57:31.112 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:57:31.112 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:57:31.112 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:57:31.112 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:57:31.112 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:57:31.112 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:57:31.113 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:57:31.113 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:57:31.113 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:57:31.113 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:57:31.113 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:57:31.113 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: main() 2024-01-13T11:57:31.113 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:57:31.113 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:57:31.113 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:57:31.113 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:57:31.113 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:57:31.113 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:57:31.113 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:57:31.113 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:57:31.114 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:57:31.114 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:57:31.114 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:57:31.114 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:57:31.114 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:57:31.114 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:57:31.114 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp2luebpqo:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpf9b7x28t:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:57:31.114 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:57:31.114 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T11:57:31.114 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: if self._apply_service(spec): 2024-01-13T11:57:31.114 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T11:57:31.114 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T11:57:31.114 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T11:57:31.114 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T11:57:31.114 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T11:57:31.115 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T11:57:31.115 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T11:57:31.115 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T11:57:31.115 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T11:57:31.115 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: raise self._value 2024-01-13T11:57:31.115 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T11:57:31.115 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: result = (True, func(*args, **kwds)) 2024-01-13T11:57:31.115 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T11:57:31.115 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: return list(map(*args)) 2024-01-13T11:57:31.115 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:57:31.115 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: return f(*arg) 2024-01-13T11:57:31.115 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:57:31.115 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:57:31.115 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:57:31.116 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: code, '\n'.join(err))) 2024-01-13T11:57:31.116 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:57:31.116 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp2luebpqo:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpf9b7x28t:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:57:31.116 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:57:31.116 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:57:31.116 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:57:31.116 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 0353b9e9-d2bc-4178-83b7-92f6a0d0c6e9 2024-01-13T11:57:31.116 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:57:31.116 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:57:31.116 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:57:31.116 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:57:31.116 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:57:31.116 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:57:31.116 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:57:31.117 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:57:31.117 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:57:31.117 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:57:31.117 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:57:31.117 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:57:31.117 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:57:31.117 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:57:31.117 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:57:31.117 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:57:31.117 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:57:31.117 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:57:31.117 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:57:31.117 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:57:31.117 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:57:31.118 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:57:31.118 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:57:31.118 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:57:31.118 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:57:31.118 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:57:31.118 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:57:31.118 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:57:31.118 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:57:31.118 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:57:31.118 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:57:31.118 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:57:31.118 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:57:31.118 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:57:31.118 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:57:31.119 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:57:31.119 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:57:31.119 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:57:31.119 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:57:31.119 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:57:31.119 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:57:31.119 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:57:31.119 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:57:31.119 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:57:31.119 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:57:31.119 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:57:31.119 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:57:31.119 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:57:31.119 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:57:31.119 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:57:31.120 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:57:31.120 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:57:31.120 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:57:31.120 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:57:31.120 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:57:31.120 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:57:31.120 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:57:31.120 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:57:31.120 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: main() 2024-01-13T11:57:31.120 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:57:31.120 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:57:31.120 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:57:31.120 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:57:31.120 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:57:31.121 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:57:31.121 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:57:31.121 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:57:31.121 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:57:31.121 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:57:31.121 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:57:31.121 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:57:31.121 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:57:31.121 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:57:31.121 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp2luebpqo:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpf9b7x28t:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:57:31.121 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: cluster 2024-01-13T11:57:29.456922+0000 mgr.smithi088.xwlxdh (mgr.14182) 353 : cluster [DBG] pgmap v325: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:31.121 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: audit 2024-01-13T11:57:29.737339+0000 mon.smithi088 (mon.0) 586 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:57:31.121 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:30 smithi088 bash[14069]: cluster 2024-01-13T11:57:30.054706+0000 mon.smithi088 (mon.0) 587 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T11:57:31.122 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: cephadm 2024-01-13T11:57:29.452238+0000 mgr.smithi088.xwlxdh (mgr.14182) 351 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:57:31.123 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:57:31.123 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:57:31.123 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: return f(*arg) 2024-01-13T11:57:31.123 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:57:31.123 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:57:31.123 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:57:31.123 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: code, '\n'.join(err))) 2024-01-13T11:57:31.123 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi088/config 2024-01-13T11:57:31.123 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpg7qhqtho:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp28z3abgb:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:57:31.123 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:57:31.123 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:57:31.123 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:57:31.123 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 9461ac01-d08d-44df-a942-4eba5cd6e3a3 2024-01-13T11:57:31.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:57:31.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:57:31.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:57:31.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:57:31.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:57:31.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:57:31.124 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:57:31.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:57:31.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:57:31.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:57:31.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:57:31.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:57:31.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:57:31.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:57:31.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:57:31.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:57:31.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:57:31.125 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:57:31.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:57:31.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:57:31.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:57:31.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:57:31.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:57:31.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:57:31.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:57:31.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:57:31.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:57:31.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:57:31.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:57:31.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:57:31.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:57:31.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:57:31.126 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:57:31.127 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:57:31.127 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:57:31.127 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:57:31.127 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:57:31.127 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:57:31.127 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:57:31.127 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:57:31.127 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:57:31.127 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:57:31.127 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:57:31.127 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:57:31.127 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:57:31.127 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:57:31.127 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:57:31.127 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:57:31.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:57:31.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:57:31.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:57:31.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:57:31.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:57:31.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:57:31.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:57:31.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:57:31.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:57:31.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:57:31.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: main() 2024-01-13T11:57:31.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:57:31.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:57:31.128 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:57:31.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:57:31.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:57:31.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:57:31.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:57:31.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:57:31.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:57:31.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:57:31.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:57:31.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:57:31.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:57:31.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:57:31.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpg7qhqtho:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp28z3abgb:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:57:31.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: cephadm 2024-01-13T11:57:29.455525+0000 mgr.smithi088.xwlxdh (mgr.14182) 352 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T11:57:31.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: service_id: all-available-devices 2024-01-13T11:57:31.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: service_name: osd.all-available-devices 2024-01-13T11:57:31.129 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: placement: 2024-01-13T11:57:31.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: host_pattern: '*' 2024-01-13T11:57:31.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: spec: 2024-01-13T11:57:31.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: data_devices: 2024-01-13T11:57:31.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: all: true 2024-01-13T11:57:31.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: filter_logic: AND 2024-01-13T11:57:31.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: objectstore: bluestore 2024-01-13T11:57:31.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:57:31.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp2luebpqo:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpf9b7x28t:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:57:31.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:57:31.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:57:31.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:57:31.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 0353b9e9-d2bc-4178-83b7-92f6a0d0c6e9 2024-01-13T11:57:31.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:57:31.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:57:31.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:57:31.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:57:31.130 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:57:31.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:57:31.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:57:31.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:57:31.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:57:31.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:57:31.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:57:31.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:57:31.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:57:31.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:57:31.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:57:31.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:57:31.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:57:31.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:57:31.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:57:31.131 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:57:31.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:57:31.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:57:31.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:57:31.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:57:31.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:57:31.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:57:31.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:57:31.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:57:31.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:57:31.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:57:31.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:57:31.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:57:31.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:57:31.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:57:31.132 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:57:31.133 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:57:31.133 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:57:31.133 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:57:31.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:57:31.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:57:31.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:57:31.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:57:31.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:57:31.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:57:31.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:57:31.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:57:31.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:57:31.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:57:31.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:57:31.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:57:31.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:57:31.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:57:31.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:57:31.134 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:57:31.135 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:57:31.135 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:57:31.135 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:57:31.135 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:57:31.135 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: main() 2024-01-13T11:57:31.135 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:57:31.135 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:57:31.135 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:57:31.135 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:57:31.135 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:57:31.135 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:57:31.135 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:57:31.135 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:57:31.135 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:57:31.135 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:57:31.135 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:57:31.136 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:57:31.136 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:57:31.136 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:57:31.136 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp2luebpqo:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpf9b7x28t:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:57:31.136 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:57:31.136 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T11:57:31.136 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: if self._apply_service(spec): 2024-01-13T11:57:31.136 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T11:57:31.136 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T11:57:31.136 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T11:57:31.136 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T11:57:31.136 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T11:57:31.136 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T11:57:31.136 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T11:57:31.137 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T11:57:31.137 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T11:57:31.137 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: raise self._value 2024-01-13T11:57:31.137 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T11:57:31.137 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: result = (True, func(*args, **kwds)) 2024-01-13T11:57:31.137 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T11:57:31.137 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: return list(map(*args)) 2024-01-13T11:57:31.137 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:57:31.137 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: return f(*arg) 2024-01-13T11:57:31.137 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:57:31.137 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:57:31.137 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:57:31.137 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: code, '\n'.join(err))) 2024-01-13T11:57:31.137 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:57:31.138 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp2luebpqo:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpf9b7x28t:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:57:31.138 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:57:31.138 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:57:31.138 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:57:31.138 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 0353b9e9-d2bc-4178-83b7-92f6a0d0c6e9 2024-01-13T11:57:31.138 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:57:31.138 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:57:31.138 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:57:31.138 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:57:31.138 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:57:31.138 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:57:31.138 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:57:31.138 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:57:31.138 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:57:31.139 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:57:31.139 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:57:31.139 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:57:31.139 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:57:31.139 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:57:31.139 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:57:31.139 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:57:31.139 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:57:31.139 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:57:31.139 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:57:31.139 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:57:31.139 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:57:31.139 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:57:31.139 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:57:31.139 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:57:31.140 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:57:31.140 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:57:31.140 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:57:31.140 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:57:31.140 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:57:31.140 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:57:31.140 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:57:31.140 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:57:31.140 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:57:31.140 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:57:31.140 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:57:31.140 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:57:31.140 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:57:31.140 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:57:31.141 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:57:31.141 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:57:31.141 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:57:31.141 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:57:31.141 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:57:31.141 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:57:31.141 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:57:31.141 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:57:31.141 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:57:31.141 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:57:31.141 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:57:31.141 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:57:31.141 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:57:31.141 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:57:31.142 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:57:31.142 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:57:31.142 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:57:31.142 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:57:31.142 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:57:31.142 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:57:31.142 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: main() 2024-01-13T11:57:31.142 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:57:31.142 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:57:31.142 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:57:31.142 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:57:31.142 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:57:31.142 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:57:31.142 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:57:31.142 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:57:31.143 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:57:31.143 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:57:31.143 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:57:31.143 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:57:31.143 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:57:31.143 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:57:31.143 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp2luebpqo:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpf9b7x28t:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:57:31.143 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: cluster 2024-01-13T11:57:29.456922+0000 mgr.smithi088.xwlxdh (mgr.14182) 353 : cluster [DBG] pgmap v325: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:31.143 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: audit 2024-01-13T11:57:29.737339+0000 mon.smithi088 (mon.0) 586 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:57:31.143 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:30 smithi138 bash[17620]: cluster 2024-01-13T11:57:30.054706+0000 mon.smithi088 (mon.0) 587 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T11:57:31.713 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":41,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:57:32.122 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:31 smithi138 bash[17620]: audit 2024-01-13T11:57:30.870995+0000 mon.smithi088 (mon.0) 588 : audit [DBG] from='client.? 172.21.15.88:0/13240744' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:57:32.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:31 smithi088 bash[14069]: audit 2024-01-13T11:57:30.870995+0000 mon.smithi088 (mon.0) 588 : audit [DBG] from='client.? 172.21.15.88:0/13240744' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:57:32.714 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:57:32.872 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:32 smithi138 bash[17620]: cluster 2024-01-13T11:57:31.457922+0000 mgr.smithi088.xwlxdh (mgr.14182) 354 : cluster [DBG] pgmap v326: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:33.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:32 smithi088 bash[14069]: cluster 2024-01-13T11:57:31.457922+0000 mgr.smithi088.xwlxdh (mgr.14182) 354 : cluster [DBG] pgmap v326: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:35.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:35 smithi138 bash[17620]: cluster 2024-01-13T11:57:33.459068+0000 mgr.smithi088.xwlxdh (mgr.14182) 355 : cluster [DBG] pgmap v327: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:35.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:35 smithi088 bash[14069]: cluster 2024-01-13T11:57:33.459068+0000 mgr.smithi088.xwlxdh (mgr.14182) 355 : cluster [DBG] pgmap v327: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:35.879 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:57:36.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:36 smithi138 bash[17620]: audit 2024-01-13T11:57:35.891153+0000 mon.smithi088 (mon.0) 589 : audit [DBG] from='client.? 172.21.15.88:0/2007513370' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:57:36.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:36 smithi088 bash[14069]: audit 2024-01-13T11:57:35.891153+0000 mon.smithi088 (mon.0) 589 : audit [DBG] from='client.? 172.21.15.88:0/2007513370' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:57:36.628 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":41,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:57:37.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:37 smithi138 bash[17620]: cluster 2024-01-13T11:57:35.460213+0000 mgr.smithi088.xwlxdh (mgr.14182) 356 : cluster [DBG] pgmap v328: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:37.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:37 smithi088 bash[14069]: cluster 2024-01-13T11:57:35.460213+0000 mgr.smithi088.xwlxdh (mgr.14182) 356 : cluster [DBG] pgmap v328: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:37.629 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:57:39.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:39 smithi138 bash[17620]: cluster 2024-01-13T11:57:37.461344+0000 mgr.smithi088.xwlxdh (mgr.14182) 357 : cluster [DBG] pgmap v329: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:39.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:39 smithi088 bash[14069]: cluster 2024-01-13T11:57:37.461344+0000 mgr.smithi088.xwlxdh (mgr.14182) 357 : cluster [DBG] pgmap v329: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:40.793 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:57:41.120 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:41 smithi088 bash[14069]: cluster 2024-01-13T11:57:39.462410+0000 mgr.smithi088.xwlxdh (mgr.14182) 358 : cluster [DBG] pgmap v330: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:41.120 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:41 smithi088 bash[14069]: audit 2024-01-13T11:57:40.804898+0000 mon.smithi088 (mon.0) 590 : audit [DBG] from='client.? 172.21.15.88:0/1271456738' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:57:41.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:41 smithi138 bash[17620]: cluster 2024-01-13T11:57:39.462410+0000 mgr.smithi088.xwlxdh (mgr.14182) 358 : cluster [DBG] pgmap v330: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:41.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:41 smithi138 bash[17620]: audit 2024-01-13T11:57:40.804898+0000 mon.smithi088 (mon.0) 590 : audit [DBG] from='client.? 172.21.15.88:0/1271456738' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:57:41.600 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":41,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:57:42.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:42 smithi138 bash[17620]: cluster 2024-01-13T11:57:41.463236+0000 mgr.smithi088.xwlxdh (mgr.14182) 359 : cluster [DBG] pgmap v331: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:42.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:42 smithi088 bash[14069]: cluster 2024-01-13T11:57:41.463236+0000 mgr.smithi088.xwlxdh (mgr.14182) 359 : cluster [DBG] pgmap v331: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:42.601 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:57:45.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:45 smithi138 bash[17620]: cluster 2024-01-13T11:57:43.463977+0000 mgr.smithi088.xwlxdh (mgr.14182) 360 : cluster [DBG] pgmap v332: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:45.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:45 smithi138 bash[17620]: audit 2024-01-13T11:57:44.739804+0000 mon.smithi088 (mon.0) 591 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:57:45.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:45 smithi088 bash[14069]: cluster 2024-01-13T11:57:43.463977+0000 mgr.smithi088.xwlxdh (mgr.14182) 360 : cluster [DBG] pgmap v332: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:45.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:45 smithi088 bash[14069]: audit 2024-01-13T11:57:44.739804+0000 mon.smithi088 (mon.0) 591 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:57:45.899 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:57:46.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:46 smithi138 bash[17620]: audit 2024-01-13T11:57:45.910900+0000 mon.smithi088 (mon.0) 592 : audit [DBG] from='client.? 172.21.15.88:0/3690181991' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:57:46.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:46 smithi088 bash[14069]: audit 2024-01-13T11:57:45.910900+0000 mon.smithi088 (mon.0) 592 : audit [DBG] from='client.? 172.21.15.88:0/3690181991' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:57:46.734 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":41,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:57:47.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:47 smithi138 bash[17620]: cluster 2024-01-13T11:57:45.464978+0000 mgr.smithi088.xwlxdh (mgr.14182) 361 : cluster [DBG] pgmap v333: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:47.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:47 smithi088 bash[14069]: cluster 2024-01-13T11:57:45.464978+0000 mgr.smithi088.xwlxdh (mgr.14182) 361 : cluster [DBG] pgmap v333: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:47.735 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:57:49.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:49 smithi088 bash[14069]: cluster 2024-01-13T11:57:47.466030+0000 mgr.smithi088.xwlxdh (mgr.14182) 362 : cluster [DBG] pgmap v334: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:49.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:49 smithi138 bash[17620]: cluster 2024-01-13T11:57:47.466030+0000 mgr.smithi088.xwlxdh (mgr.14182) 362 : cluster [DBG] pgmap v334: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:50.829 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:57:51.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:51 smithi088 bash[14069]: cluster 2024-01-13T11:57:49.467078+0000 mgr.smithi088.xwlxdh (mgr.14182) 363 : cluster [DBG] pgmap v335: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:51.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:51 smithi088 bash[14069]: audit 2024-01-13T11:57:50.841057+0000 mon.smithi088 (mon.0) 593 : audit [DBG] from='client.? 172.21.15.88:0/2698870378' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:57:51.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:51 smithi138 bash[17620]: cluster 2024-01-13T11:57:49.467078+0000 mgr.smithi088.xwlxdh (mgr.14182) 363 : cluster [DBG] pgmap v335: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:51.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:51 smithi138 bash[17620]: audit 2024-01-13T11:57:50.841057+0000 mon.smithi088 (mon.0) 593 : audit [DBG] from='client.? 172.21.15.88:0/2698870378' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:57:51.683 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":41,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:57:52.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:52 smithi138 bash[17620]: cluster 2024-01-13T11:57:51.468073+0000 mgr.smithi088.xwlxdh (mgr.14182) 364 : cluster [DBG] pgmap v336: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:52.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:52 smithi088 bash[14069]: cluster 2024-01-13T11:57:51.468073+0000 mgr.smithi088.xwlxdh (mgr.14182) 364 : cluster [DBG] pgmap v336: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:52.684 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:57:55.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:55 smithi088 bash[14069]: cluster 2024-01-13T11:57:53.469142+0000 mgr.smithi088.xwlxdh (mgr.14182) 365 : cluster [DBG] pgmap v337: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:55.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:55 smithi138 bash[17620]: cluster 2024-01-13T11:57:53.469142+0000 mgr.smithi088.xwlxdh (mgr.14182) 365 : cluster [DBG] pgmap v337: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:55.985 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:57:56.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:56 smithi138 bash[17620]: audit 2024-01-13T11:57:55.997577+0000 mon.smithi088 (mon.0) 594 : audit [DBG] from='client.? 172.21.15.88:0/2799719079' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:57:56.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:56 smithi088 bash[14069]: audit 2024-01-13T11:57:55.997577+0000 mon.smithi088 (mon.0) 594 : audit [DBG] from='client.? 172.21.15.88:0/2799719079' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:57:56.783 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":41,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:57:57.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:57 smithi088 bash[14069]: cluster 2024-01-13T11:57:55.470126+0000 mgr.smithi088.xwlxdh (mgr.14182) 366 : cluster [DBG] pgmap v338: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:57.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:57 smithi138 bash[17620]: cluster 2024-01-13T11:57:55.470126+0000 mgr.smithi088.xwlxdh (mgr.14182) 366 : cluster [DBG] pgmap v338: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:57.784 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:57:59.318 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:57:59 smithi088 bash[14069]: cluster 2024-01-13T11:57:57.471207+0000 mgr.smithi088.xwlxdh (mgr.14182) 367 : cluster [DBG] pgmap v339: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:57:59.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:57:59 smithi138 bash[17620]: cluster 2024-01-13T11:57:57.471207+0000 mgr.smithi088.xwlxdh (mgr.14182) 367 : cluster [DBG] pgmap v339: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:01.120 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:58:01.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:01 smithi138 bash[17620]: cluster 2024-01-13T11:57:59.471965+0000 mgr.smithi088.xwlxdh (mgr.14182) 368 : cluster [DBG] pgmap v340: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:01.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:01 smithi088 bash[14069]: cluster 2024-01-13T11:57:59.471965+0000 mgr.smithi088.xwlxdh (mgr.14182) 368 : cluster [DBG] pgmap v340: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:01.963 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":41,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:58:02.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:02 smithi138 bash[17620]: audit 2024-01-13T11:58:01.132432+0000 mon.smithi088 (mon.0) 595 : audit [DBG] from='client.? 172.21.15.88:0/2083482459' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:58:02.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:02 smithi138 bash[17620]: cluster 2024-01-13T11:58:01.472502+0000 mgr.smithi088.xwlxdh (mgr.14182) 369 : cluster [DBG] pgmap v341: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:02.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:02 smithi088 bash[14069]: audit 2024-01-13T11:58:01.132432+0000 mon.smithi088 (mon.0) 595 : audit [DBG] from='client.? 172.21.15.88:0/2083482459' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:58:02.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:02 smithi088 bash[14069]: cluster 2024-01-13T11:58:01.472502+0000 mgr.smithi088.xwlxdh (mgr.14182) 369 : cluster [DBG] pgmap v341: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:02.965 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:58:05.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:05 smithi138 bash[17620]: cluster 2024-01-13T11:58:03.473090+0000 mgr.smithi088.xwlxdh (mgr.14182) 370 : cluster [DBG] pgmap v342: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:05.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:05 smithi088 bash[14069]: cluster 2024-01-13T11:58:03.473090+0000 mgr.smithi088.xwlxdh (mgr.14182) 370 : cluster [DBG] pgmap v342: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:06.405 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:58:07.171 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":41,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:58:07.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:07 smithi138 bash[17620]: cluster 2024-01-13T11:58:05.473825+0000 mgr.smithi088.xwlxdh (mgr.14182) 371 : cluster [DBG] pgmap v343: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:07.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:07 smithi138 bash[17620]: audit 2024-01-13T11:58:06.417708+0000 mon.smithi088 (mon.0) 596 : audit [DBG] from='client.? 172.21.15.88:0/2721031152' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:58:07.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:07 smithi088 bash[14069]: cluster 2024-01-13T11:58:05.473825+0000 mgr.smithi088.xwlxdh (mgr.14182) 371 : cluster [DBG] pgmap v343: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:07.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:07 smithi088 bash[14069]: audit 2024-01-13T11:58:06.417708+0000 mon.smithi088 (mon.0) 596 : audit [DBG] from='client.? 172.21.15.88:0/2721031152' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:58:08.172 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:58:09.267 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:09 smithi088 bash[14069]: cluster 2024-01-13T11:58:07.474834+0000 mgr.smithi088.xwlxdh (mgr.14182) 372 : cluster [DBG] pgmap v344: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:09.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:09 smithi138 bash[17620]: cluster 2024-01-13T11:58:07.474834+0000 mgr.smithi088.xwlxdh (mgr.14182) 372 : cluster [DBG] pgmap v344: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:11.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:11 smithi138 bash[17620]: cluster 2024-01-13T11:58:09.475404+0000 mgr.smithi088.xwlxdh (mgr.14182) 373 : cluster [DBG] pgmap v345: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:11.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:11 smithi088 bash[14069]: cluster 2024-01-13T11:58:09.475404+0000 mgr.smithi088.xwlxdh (mgr.14182) 373 : cluster [DBG] pgmap v345: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:11.425 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:58:12.267 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":41,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:58:12.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:12 smithi138 bash[17620]: audit 2024-01-13T11:58:11.436927+0000 mon.smithi088 (mon.0) 597 : audit [DBG] from='client.? 172.21.15.88:0/3835729551' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:58:12.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:12 smithi138 bash[17620]: cluster 2024-01-13T11:58:11.476373+0000 mgr.smithi088.xwlxdh (mgr.14182) 374 : cluster [DBG] pgmap v346: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:12.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:12 smithi088 bash[14069]: audit 2024-01-13T11:58:11.436927+0000 mon.smithi088 (mon.0) 597 : audit [DBG] from='client.? 172.21.15.88:0/3835729551' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:58:12.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:12 smithi088 bash[14069]: cluster 2024-01-13T11:58:11.476373+0000 mgr.smithi088.xwlxdh (mgr.14182) 374 : cluster [DBG] pgmap v346: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:13.269 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:58:15.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:15 smithi088 bash[14069]: cluster 2024-01-13T11:58:13.476972+0000 mgr.smithi088.xwlxdh (mgr.14182) 375 : cluster [DBG] pgmap v347: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:15.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:15 smithi138 bash[17620]: cluster 2024-01-13T11:58:13.476972+0000 mgr.smithi088.xwlxdh (mgr.14182) 375 : cluster [DBG] pgmap v347: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:16.669 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:58:17.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:17 smithi138 bash[17620]: cluster 2024-01-13T11:58:15.477972+0000 mgr.smithi088.xwlxdh (mgr.14182) 376 : cluster [DBG] pgmap v348: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:17.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:17 smithi138 bash[17620]: audit 2024-01-13T11:58:16.681313+0000 mon.smithi088 (mon.0) 598 : audit [DBG] from='client.? 172.21.15.88:0/2051996591' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:58:17.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:17 smithi088 bash[14069]: cluster 2024-01-13T11:58:15.477972+0000 mgr.smithi088.xwlxdh (mgr.14182) 376 : cluster [DBG] pgmap v348: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:17.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:17 smithi088 bash[14069]: audit 2024-01-13T11:58:16.681313+0000 mon.smithi088 (mon.0) 598 : audit [DBG] from='client.? 172.21.15.88:0/2051996591' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:58:17.459 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":41,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:58:18.460 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:58:19.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:19 smithi138 bash[17620]: cluster 2024-01-13T11:58:17.478993+0000 mgr.smithi088.xwlxdh (mgr.14182) 377 : cluster [DBG] pgmap v349: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:19.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:19 smithi088 bash[14069]: cluster 2024-01-13T11:58:17.478993+0000 mgr.smithi088.xwlxdh (mgr.14182) 377 : cluster [DBG] pgmap v349: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:21.299 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:21 smithi088 bash[14069]: cluster 2024-01-13T11:58:19.479928+0000 mgr.smithi088.xwlxdh (mgr.14182) 378 : cluster [DBG] pgmap v350: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:21.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:21 smithi138 bash[17620]: cluster 2024-01-13T11:58:19.479928+0000 mgr.smithi088.xwlxdh (mgr.14182) 378 : cluster [DBG] pgmap v350: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:21.683 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:58:22.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:22 smithi138 bash[17620]: cluster 2024-01-13T11:58:21.480450+0000 mgr.smithi088.xwlxdh (mgr.14182) 379 : cluster [DBG] pgmap v351: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:22.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:22 smithi138 bash[17620]: audit 2024-01-13T11:58:21.695669+0000 mon.smithi088 (mon.0) 599 : audit [DBG] from='client.? 172.21.15.88:0/354153223' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:58:22.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:22 smithi088 bash[14069]: cluster 2024-01-13T11:58:21.480450+0000 mgr.smithi088.xwlxdh (mgr.14182) 379 : cluster [DBG] pgmap v351: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:22.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:22 smithi088 bash[14069]: audit 2024-01-13T11:58:21.695669+0000 mon.smithi088 (mon.0) 599 : audit [DBG] from='client.? 172.21.15.88:0/354153223' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:58:22.572 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":41,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:58:23.574 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:58:25.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:25 smithi138 bash[17620]: cluster 2024-01-13T11:58:23.481474+0000 mgr.smithi088.xwlxdh (mgr.14182) 380 : cluster [DBG] pgmap v352: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:25.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:25 smithi088 bash[14069]: cluster 2024-01-13T11:58:23.481474+0000 mgr.smithi088.xwlxdh (mgr.14182) 380 : cluster [DBG] pgmap v352: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:26.811 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:58:27.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:27 smithi088 bash[14069]: cluster 2024-01-13T11:58:25.482506+0000 mgr.smithi088.xwlxdh (mgr.14182) 381 : cluster [DBG] pgmap v353: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:27.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:27 smithi088 bash[14069]: audit 2024-01-13T11:58:26.823808+0000 mon.smithi088 (mon.0) 600 : audit [DBG] from='client.? 172.21.15.88:0/917183604' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:58:27.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:27 smithi138 bash[17620]: cluster 2024-01-13T11:58:25.482506+0000 mgr.smithi088.xwlxdh (mgr.14182) 381 : cluster [DBG] pgmap v353: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:27.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:27 smithi138 bash[17620]: audit 2024-01-13T11:58:26.823808+0000 mon.smithi088 (mon.0) 600 : audit [DBG] from='client.? 172.21.15.88:0/917183604' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:58:27.560 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":41,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:58:28.561 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:58:29.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:29 smithi138 bash[17620]: cluster 2024-01-13T11:58:27.483447+0000 mgr.smithi088.xwlxdh (mgr.14182) 382 : cluster [DBG] pgmap v354: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:29.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:29 smithi088 bash[14069]: cluster 2024-01-13T11:58:27.483447+0000 mgr.smithi088.xwlxdh (mgr.14182) 382 : cluster [DBG] pgmap v354: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:30.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:30 smithi088 bash[14069]: audit 2024-01-13T11:58:29.459391+0000 mon.smithi088 (mon.0) 601 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:58:30.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:30 smithi088 bash[14069]: audit 2024-01-13T11:58:29.731705+0000 mon.smithi088 (mon.0) 602 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:58:30.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:30 smithi088 bash[14069]: audit 2024-01-13T11:58:29.754951+0000 mon.smithi088 (mon.0) 603 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config rm", "who": "osd/host:smithi088", "name": "osd_memory_target"}]: dispatch 2024-01-13T11:58:30.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:30 smithi138 bash[17620]: audit 2024-01-13T11:58:29.459391+0000 mon.smithi088 (mon.0) 601 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T11:58:30.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:30 smithi138 bash[17620]: audit 2024-01-13T11:58:29.731705+0000 mon.smithi088 (mon.0) 602 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:58:30.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:30 smithi138 bash[17620]: audit 2024-01-13T11:58:29.754951+0000 mon.smithi088 (mon.0) 603 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config rm", "who": "osd/host:smithi088", "name": "osd_memory_target"}]: dispatch 2024-01-13T11:58:31.120 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:31 smithi088 bash[14069]: cluster 2024-01-13T11:58:29.484010+0000 mgr.smithi088.xwlxdh (mgr.14182) 383 : cluster [DBG] pgmap v355: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:31.120 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:31 smithi088 bash[14069]: audit 2024-01-13T11:58:30.037127+0000 mon.smithi088 (mon.0) 604 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config rm", "who": "osd/host:smithi138", "name": "osd_memory_target"}]: dispatch 2024-01-13T11:58:31.120 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:31 smithi088 bash[14069]: audit 2024-01-13T11:58:30.038679+0000 mon.smithi088 (mon.0) 605 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:58:31.120 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:31 smithi088 bash[14069]: audit 2024-01-13T11:58:30.039910+0000 mon.smithi088 (mon.0) 606 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T11:58:31.120 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:31 smithi088 bash[14069]: audit 2024-01-13T11:58:30.049058+0000 mon.smithi088 (mon.0) 607 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:58:31.120 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:31 smithi088 bash[14069]: cluster 2024-01-13T11:58:30.049668+0000 mgr.smithi088.xwlxdh (mgr.14182) 384 : cluster [DBG] pgmap v356: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:31.121 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:31 smithi088 bash[14069]: audit 2024-01-13T11:58:30.057156+0000 mon.smithi088 (mon.0) 608 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T11:58:31.121 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:31 smithi088 bash[14069]: audit 2024-01-13T11:58:30.064115+0000 mon.smithi088 (mon.0) 609 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:58:31.121 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:31 smithi088 bash[14069]: audit 2024-01-13T11:58:30.066502+0000 mon.smithi088 (mon.0) 610 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:58:31.121 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:31 smithi088 bash[14069]: audit 2024-01-13T11:58:30.066988+0000 mon.smithi088 (mon.0) 611 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:58:31.121 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:31 smithi088 bash[14069]: audit 2024-01-13T11:58:30.068077+0000 mon.smithi088 (mon.0) 612 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:58:31.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:31 smithi138 bash[17620]: cluster 2024-01-13T11:58:29.484010+0000 mgr.smithi088.xwlxdh (mgr.14182) 383 : cluster [DBG] pgmap v355: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:31.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:31 smithi138 bash[17620]: audit 2024-01-13T11:58:30.037127+0000 mon.smithi088 (mon.0) 604 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config rm", "who": "osd/host:smithi138", "name": "osd_memory_target"}]: dispatch 2024-01-13T11:58:31.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:31 smithi138 bash[17620]: audit 2024-01-13T11:58:30.038679+0000 mon.smithi088 (mon.0) 605 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:58:31.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:31 smithi138 bash[17620]: audit 2024-01-13T11:58:30.039910+0000 mon.smithi088 (mon.0) 606 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T11:58:31.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:31 smithi138 bash[17620]: audit 2024-01-13T11:58:30.049058+0000 mon.smithi088 (mon.0) 607 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' 2024-01-13T11:58:31.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:31 smithi138 bash[17620]: cluster 2024-01-13T11:58:30.049668+0000 mgr.smithi088.xwlxdh (mgr.14182) 384 : cluster [DBG] pgmap v356: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:31.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:31 smithi138 bash[17620]: audit 2024-01-13T11:58:30.057156+0000 mon.smithi088 (mon.0) 608 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T11:58:31.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:31 smithi138 bash[17620]: audit 2024-01-13T11:58:30.064115+0000 mon.smithi088 (mon.0) 609 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:58:31.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:31 smithi138 bash[17620]: audit 2024-01-13T11:58:30.066502+0000 mon.smithi088 (mon.0) 610 : audit [INF] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T11:58:31.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:31 smithi138 bash[17620]: audit 2024-01-13T11:58:30.066988+0000 mon.smithi088 (mon.0) 611 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:58:31.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:31 smithi138 bash[17620]: audit 2024-01-13T11:58:30.068077+0000 mon.smithi088 (mon.0) 612 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T11:58:32.111 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:58:32.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:32 smithi138 bash[17620]: cluster 2024-01-13T11:58:31.046534+0000 mon.smithi088 (mon.0) 613 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T11:58:32.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:32 smithi138 bash[17620]: cluster 2024-01-13T11:58:31.046562+0000 mon.smithi088 (mon.0) 614 : cluster [INF] Cluster is now healthy 2024-01-13T11:58:32.385 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:32 smithi088 bash[14069]: cluster 2024-01-13T11:58:31.046534+0000 mon.smithi088 (mon.0) 613 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T11:58:32.385 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:32 smithi088 bash[14069]: cluster 2024-01-13T11:58:31.046562+0000 mon.smithi088 (mon.0) 614 : cluster [INF] Cluster is now healthy 2024-01-13T11:58:32.953 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":41,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:58:33.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:33 smithi138 bash[17620]: cluster 2024-01-13T11:58:32.050092+0000 mgr.smithi088.xwlxdh (mgr.14182) 385 : cluster [DBG] pgmap v357: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:33.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:33 smithi138 bash[17620]: audit 2024-01-13T11:58:32.123691+0000 mon.smithi088 (mon.0) 615 : audit [DBG] from='client.? 172.21.15.88:0/1669805488' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:58:33.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:33 smithi088 bash[14069]: cluster 2024-01-13T11:58:32.050092+0000 mgr.smithi088.xwlxdh (mgr.14182) 385 : cluster [DBG] pgmap v357: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:33.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:33 smithi088 bash[14069]: audit 2024-01-13T11:58:32.123691+0000 mon.smithi088 (mon.0) 615 : audit [DBG] from='client.? 172.21.15.88:0/1669805488' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:58:33.954 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:58:35.122 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:35 smithi138 bash[17620]: cluster 2024-01-13T11:58:34.050557+0000 mgr.smithi088.xwlxdh (mgr.14182) 386 : cluster [DBG] pgmap v358: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:35.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:35 smithi088 bash[14069]: cluster 2024-01-13T11:58:34.050557+0000 mgr.smithi088.xwlxdh (mgr.14182) 386 : cluster [DBG] pgmap v358: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:37.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:37 smithi138 bash[17620]: cluster 2024-01-13T11:58:36.050848+0000 mgr.smithi088.xwlxdh (mgr.14182) 387 : cluster [DBG] pgmap v359: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:37.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:37 smithi088 bash[14069]: cluster 2024-01-13T11:58:36.050848+0000 mgr.smithi088.xwlxdh (mgr.14182) 387 : cluster [DBG] pgmap v359: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:37.816 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:58:38.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:38 smithi138 bash[17620]: audit 2024-01-13T11:58:37.828860+0000 mon.smithi088 (mon.0) 616 : audit [DBG] from='client.? 172.21.15.88:0/4130719437' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:58:38.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:38 smithi088 bash[14069]: audit 2024-01-13T11:58:37.828860+0000 mon.smithi088 (mon.0) 616 : audit [DBG] from='client.? 172.21.15.88:0/4130719437' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:58:39.341 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:39 smithi088 bash[14069]: cluster 2024-01-13T11:58:38.051216+0000 mgr.smithi088.xwlxdh (mgr.14182) 388 : cluster [DBG] pgmap v360: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:39.341 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:39 smithi088 bash[14069]: audit 2024-01-13T11:58:38.835996+0000 mon.smithi138 (mon.1) 19 : audit [INF] from='client.? 172.21.15.138:0/2722453417' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a27e472f-fdc4-4562-8f46-ffc6a8bda612"}]: dispatch 2024-01-13T11:58:39.341 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:39 smithi088 bash[14069]: audit 2024-01-13T11:58:38.836240+0000 mon.smithi088 (mon.0) 617 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a27e472f-fdc4-4562-8f46-ffc6a8bda612"}]: dispatch 2024-01-13T11:58:39.341 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:39 smithi088 bash[14069]: audit 2024-01-13T11:58:38.843175+0000 mon.smithi088 (mon.0) 618 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a27e472f-fdc4-4562-8f46-ffc6a8bda612"}]': finished 2024-01-13T11:58:39.341 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:39 smithi088 bash[14069]: cluster 2024-01-13T11:58:38.843309+0000 mon.smithi088 (mon.0) 619 : cluster [DBG] osdmap e42: 1 total, 0 up, 1 in 2024-01-13T11:58:39.341 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:39 smithi088 bash[14069]: audit 2024-01-13T11:58:38.843550+0000 mon.smithi088 (mon.0) 620 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:58:39.342 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":41,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:58:39.362 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:39 smithi138 bash[17620]: cluster 2024-01-13T11:58:38.051216+0000 mgr.smithi088.xwlxdh (mgr.14182) 388 : cluster [DBG] pgmap v360: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:39.362 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:39 smithi138 bash[17620]: audit 2024-01-13T11:58:38.835996+0000 mon.smithi138 (mon.1) 19 : audit [INF] from='client.? 172.21.15.138:0/2722453417' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a27e472f-fdc4-4562-8f46-ffc6a8bda612"}]: dispatch 2024-01-13T11:58:39.362 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:39 smithi138 bash[17620]: audit 2024-01-13T11:58:38.836240+0000 mon.smithi088 (mon.0) 617 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a27e472f-fdc4-4562-8f46-ffc6a8bda612"}]: dispatch 2024-01-13T11:58:39.362 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:39 smithi138 bash[17620]: audit 2024-01-13T11:58:38.843175+0000 mon.smithi088 (mon.0) 618 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a27e472f-fdc4-4562-8f46-ffc6a8bda612"}]': finished 2024-01-13T11:58:39.362 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:39 smithi138 bash[17620]: cluster 2024-01-13T11:58:38.843309+0000 mon.smithi088 (mon.0) 619 : cluster [DBG] osdmap e42: 1 total, 0 up, 1 in 2024-01-13T11:58:39.362 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:39 smithi138 bash[17620]: audit 2024-01-13T11:58:38.843550+0000 mon.smithi088 (mon.0) 620 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:58:40.343 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:58:40.355 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:40 smithi088 bash[14069]: audit 2024-01-13T11:58:39.195116+0000 mon.smithi138 (mon.1) 20 : audit [INF] from='client.? 172.21.15.138:0/2154900981' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:58:40.355 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:40 smithi088 bash[14069]: audit 2024-01-13T11:58:39.195426+0000 mon.smithi088 (mon.0) 621 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:58:40.355 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:40 smithi088 bash[14069]: audit 2024-01-13T11:58:39.200872+0000 mon.smithi088 (mon.0) 622 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:58:40.355 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:40 smithi088 bash[14069]: cluster 2024-01-13T11:58:39.200932+0000 mon.smithi088 (mon.0) 623 : cluster [DBG] osdmap e43: 0 total, 0 up, 0 in 2024-01-13T11:58:40.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:40 smithi138 bash[17620]: audit 2024-01-13T11:58:39.195116+0000 mon.smithi138 (mon.1) 20 : audit [INF] from='client.? 172.21.15.138:0/2154900981' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:58:40.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:40 smithi138 bash[17620]: audit 2024-01-13T11:58:39.195426+0000 mon.smithi088 (mon.0) 621 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:58:40.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:40 smithi138 bash[17620]: audit 2024-01-13T11:58:39.200872+0000 mon.smithi088 (mon.0) 622 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:58:40.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:40 smithi138 bash[17620]: cluster 2024-01-13T11:58:39.200932+0000 mon.smithi088 (mon.0) 623 : cluster [DBG] osdmap e43: 0 total, 0 up, 0 in 2024-01-13T11:58:41.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: cluster 2024-01-13T11:58:40.051509+0000 mgr.smithi088.xwlxdh (mgr.14182) 389 : cluster [DBG] pgmap v363: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:41.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: cephadm 2024-01-13T11:58:40.072658+0000 mgr.smithi088.xwlxdh (mgr.14182) 390 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:58:41.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:58:41.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:58:41.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: return f(*arg) 2024-01-13T11:58:41.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:58:41.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:58:41.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:58:41.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: code, '\n'.join(err))) 2024-01-13T11:58:41.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:58:41.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmppmk7d5b0:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpiz5l5df0:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:58:41.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:58:41.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:58:41.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:58:41.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new a27e472f-fdc4-4562-8f46-ffc6a8bda612 2024-01-13T11:58:41.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:58:41.391 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:58:41.392 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:58:41.392 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:58:41.392 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:58:41.392 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:58:41.392 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:58:41.392 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:58:41.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:58:41.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:58:41.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:58:41.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:58:41.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:58:41.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:58:41.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:58:41.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:58:41.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:58:41.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:58:41.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:58:41.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:58:41.393 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:58:41.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:58:41.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:58:41.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:58:41.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:58:41.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:58:41.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:58:41.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:58:41.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:58:41.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:58:41.394 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:58:41.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:58:41.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:58:41.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:58:41.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:58:41.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:58:41.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:58:41.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:58:41.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:58:41.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:58:41.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:58:41.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:58:41.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:58:41.395 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:58:41.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:58:41.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:58:41.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:58:41.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:58:41.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:58:41.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:58:41.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:58:41.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:58:41.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:58:41.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:58:41.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:58:41.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:58:41.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:58:41.396 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:58:41.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: main() 2024-01-13T11:58:41.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:58:41.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:58:41.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:58:41.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:58:41.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:58:41.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:58:41.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:58:41.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:58:41.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:58:41.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:58:41.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:58:41.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:58:41.397 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:58:41.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:58:41.398 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:41 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmppmk7d5b0:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpiz5l5df0:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:58:41.622 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: cluster 2024-01-13T11:58:40.051509+0000 mgr.smithi088.xwlxdh (mgr.14182) 389 : cluster [DBG] pgmap v363: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:41.623 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: cephadm 2024-01-13T11:58:40.072658+0000 mgr.smithi088.xwlxdh (mgr.14182) 390 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:58:41.623 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:58:41.623 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:58:41.623 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: return f(*arg) 2024-01-13T11:58:41.623 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:58:41.623 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:58:41.623 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:58:41.623 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: code, '\n'.join(err))) 2024-01-13T11:58:41.623 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:58:41.623 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmppmk7d5b0:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpiz5l5df0:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:58:41.623 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:58:41.623 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:58:41.624 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:58:41.624 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new a27e472f-fdc4-4562-8f46-ffc6a8bda612 2024-01-13T11:58:41.624 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:58:41.624 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:58:41.624 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:58:41.624 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:58:41.624 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:58:41.624 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:58:41.625 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:58:41.625 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:58:41.625 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:58:41.625 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:58:41.625 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:58:41.625 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:58:41.625 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:58:41.625 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:58:41.625 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:58:41.625 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:58:41.625 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:58:41.625 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:58:41.626 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:58:41.626 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:58:41.626 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:58:41.626 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:58:41.626 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:58:41.626 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:58:41.626 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:58:41.626 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:58:41.626 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:58:41.626 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:58:41.626 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:58:41.626 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:58:41.626 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:58:41.626 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:58:41.626 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:58:41.627 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:58:41.627 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:58:41.627 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:58:41.627 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:58:41.627 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:58:41.627 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:58:41.627 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:58:41.627 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:58:41.627 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:58:41.627 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:58:41.627 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:58:41.627 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:58:41.627 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:58:41.627 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:58:41.628 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:58:41.628 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:58:41.628 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:58:41.628 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:58:41.628 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:58:41.628 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:58:41.628 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:58:41.628 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:58:41.628 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:58:41.628 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:58:41.628 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:58:41.628 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: main() 2024-01-13T11:58:41.628 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:58:41.628 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:58:41.629 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:58:41.629 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:58:41.629 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:58:41.629 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:58:41.629 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:58:41.629 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:58:41.629 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:58:41.629 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:58:41.629 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:58:41.629 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:58:41.629 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:58:41.629 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:58:41.629 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:41 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmppmk7d5b0:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpiz5l5df0:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:58:42.418 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:42 smithi088 bash[14069]: audit 2024-01-13T11:58:41.317106+0000 mon.smithi088 (mon.0) 624 : audit [INF] from='client.? 172.21.15.88:0/3114820715' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43f58b53-2f7a-4c38-8699-eef48a293ad3"}]: dispatch 2024-01-13T11:58:42.418 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:42 smithi088 bash[14069]: audit 2024-01-13T11:58:41.322313+0000 mon.smithi088 (mon.0) 625 : audit [INF] from='client.? 172.21.15.88:0/3114820715' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "43f58b53-2f7a-4c38-8699-eef48a293ad3"}]': finished 2024-01-13T11:58:42.419 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:42 smithi088 bash[14069]: cluster 2024-01-13T11:58:41.322360+0000 mon.smithi088 (mon.0) 626 : cluster [DBG] osdmap e44: 1 total, 0 up, 1 in 2024-01-13T11:58:42.419 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:42 smithi088 bash[14069]: audit 2024-01-13T11:58:41.322553+0000 mon.smithi088 (mon.0) 627 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:58:42.419 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:42 smithi088 bash[14069]: audit 2024-01-13T11:58:41.664995+0000 mon.smithi088 (mon.0) 628 : audit [INF] from='client.? 172.21.15.88:0/3574497397' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:58:42.419 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:42 smithi088 bash[14069]: audit 2024-01-13T11:58:41.671459+0000 mon.smithi088 (mon.0) 629 : audit [INF] from='client.? 172.21.15.88:0/3574497397' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:58:42.419 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:42 smithi088 bash[14069]: cluster 2024-01-13T11:58:41.671527+0000 mon.smithi088 (mon.0) 630 : cluster [DBG] osdmap e45: 0 total, 0 up, 0 in 2024-01-13T11:58:42.591 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:42 smithi138 bash[17620]: audit 2024-01-13T11:58:41.317106+0000 mon.smithi088 (mon.0) 624 : audit [INF] from='client.? 172.21.15.88:0/3114820715' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43f58b53-2f7a-4c38-8699-eef48a293ad3"}]: dispatch 2024-01-13T11:58:42.591 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:42 smithi138 bash[17620]: audit 2024-01-13T11:58:41.322313+0000 mon.smithi088 (mon.0) 625 : audit [INF] from='client.? 172.21.15.88:0/3114820715' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "43f58b53-2f7a-4c38-8699-eef48a293ad3"}]': finished 2024-01-13T11:58:42.591 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:42 smithi138 bash[17620]: cluster 2024-01-13T11:58:41.322360+0000 mon.smithi088 (mon.0) 626 : cluster [DBG] osdmap e44: 1 total, 0 up, 1 in 2024-01-13T11:58:42.591 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:42 smithi138 bash[17620]: audit 2024-01-13T11:58:41.322553+0000 mon.smithi088 (mon.0) 627 : audit [DBG] from='mgr.14182 172.21.15.88:0/809631451' entity='mgr.smithi088.xwlxdh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T11:58:42.591 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:42 smithi138 bash[17620]: audit 2024-01-13T11:58:41.664995+0000 mon.smithi088 (mon.0) 628 : audit [INF] from='client.? 172.21.15.88:0/3574497397' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T11:58:42.592 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:42 smithi138 bash[17620]: audit 2024-01-13T11:58:41.671459+0000 mon.smithi088 (mon.0) 629 : audit [INF] from='client.? 172.21.15.88:0/3574497397' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T11:58:42.592 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:42 smithi138 bash[17620]: cluster 2024-01-13T11:58:41.671527+0000 mon.smithi088 (mon.0) 630 : cluster [DBG] osdmap e45: 0 total, 0 up, 0 in 2024-01-13T11:58:44.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:44 smithi138 bash[17620]: cluster 2024-01-13T11:58:42.051978+0000 mgr.smithi088.xwlxdh (mgr.14182) 391 : cluster [DBG] pgmap v366: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:44.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:44 smithi088 bash[14069]: cluster 2024-01-13T11:58:42.051978+0000 mgr.smithi088.xwlxdh (mgr.14182) 391 : cluster [DBG] pgmap v366: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:44.459 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:58:45.291 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: cephadm 2024-01-13T11:58:43.211412+0000 mgr.smithi088.xwlxdh (mgr.14182) 392 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:58:45.291 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:58:45.291 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:58:45.291 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: return f(*arg) 2024-01-13T11:58:45.291 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:58:45.291 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:58:45.292 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:58:45.292 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: code, '\n'.join(err))) 2024-01-13T11:58:45.292 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi088/config 2024-01-13T11:58:45.292 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpzpci51f4:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpu11gzpgj:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:58:45.292 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:58:45.292 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:58:45.292 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:58:45.292 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 43f58b53-2f7a-4c38-8699-eef48a293ad3 2024-01-13T11:58:45.292 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:58:45.292 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:58:45.292 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:58:45.293 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:58:45.293 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:58:45.293 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:58:45.293 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:58:45.294 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":45,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:58:45.296 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:58:45.296 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:58:45.296 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:58:45.296 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:58:45.296 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:58:45.296 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:58:45.296 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:58:45.297 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:58:45.297 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:58:45.297 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:58:45.297 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:58:45.297 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:58:45.297 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:58:45.297 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:58:45.297 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:58:45.297 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:58:45.297 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:58:45.297 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:58:45.297 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:58:45.298 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:58:45.298 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:58:45.298 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:58:45.298 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:58:45.298 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:58:45.298 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:58:45.298 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:58:45.298 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:58:45.298 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:58:45.298 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:58:45.298 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:58:45.299 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:58:45.299 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:58:45.299 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:58:45.299 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:58:45.299 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:58:45.299 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:58:45.299 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:58:45.299 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:58:45.299 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:58:45.299 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:58:45.299 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:58:45.299 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:58:45.300 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:58:45.300 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:58:45.300 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:58:45.300 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:58:45.300 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:58:45.300 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:58:45.300 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:58:45.300 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:58:45.300 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:58:45.300 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: main() 2024-01-13T11:58:45.300 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:58:45.301 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:58:45.301 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:58:45.301 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:58:45.301 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:58:45.301 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:58:45.301 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:58:45.301 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:58:45.301 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:58:45.301 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:58:45.301 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:58:45.301 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:58:45.302 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:58:45.302 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:58:45.302 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpzpci51f4:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpu11gzpgj:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:58:45.302 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: cephadm 2024-01-13T11:58:43.214979+0000 mgr.smithi088.xwlxdh (mgr.14182) 393 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T11:58:45.302 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: service_id: all-available-devices 2024-01-13T11:58:45.302 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: service_name: osd.all-available-devices 2024-01-13T11:58:45.302 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: placement: 2024-01-13T11:58:45.302 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: host_pattern: '*' 2024-01-13T11:58:45.302 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: spec: 2024-01-13T11:58:45.302 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: data_devices: 2024-01-13T11:58:45.302 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: all: true 2024-01-13T11:58:45.303 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: filter_logic: AND 2024-01-13T11:58:45.303 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: objectstore: bluestore 2024-01-13T11:58:45.303 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:58:45.303 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmppmk7d5b0:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpiz5l5df0:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:58:45.303 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:58:45.303 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:58:45.303 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:58:45.303 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new a27e472f-fdc4-4562-8f46-ffc6a8bda612 2024-01-13T11:58:45.303 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:58:45.303 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:58:45.303 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:58:45.303 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:58:45.304 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:58:45.304 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:58:45.304 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:58:45.304 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:58:45.304 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:58:45.304 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:58:45.304 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:58:45.304 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:58:45.304 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:58:45.304 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:58:45.304 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:58:45.305 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:58:45.305 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:58:45.305 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:58:45.305 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:58:45.305 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:58:45.305 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:58:45.305 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:58:45.305 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:58:45.305 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:58:45.305 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:58:45.305 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:58:45.305 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:58:45.305 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:58:45.306 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:58:45.306 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:58:45.306 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:58:45.306 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:58:45.306 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:58:45.306 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:58:45.306 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:58:45.306 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:58:45.306 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:58:45.306 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:58:45.306 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:58:45.307 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:58:45.307 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:58:45.307 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:58:45.307 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:58:45.307 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:58:45.307 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:58:45.307 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:58:45.307 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:58:45.307 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:58:45.307 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:58:45.307 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:58:45.307 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:58:45.308 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:58:45.308 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:58:45.308 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:58:45.308 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:58:45.308 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:58:45.308 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:58:45.308 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:58:45.308 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: main() 2024-01-13T11:58:45.308 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:58:45.308 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:58:45.308 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:58:45.309 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:58:45.309 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:58:45.309 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:58:45.309 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:58:45.309 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:58:45.309 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:58:45.309 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:58:45.309 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:58:45.309 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:58:45.309 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:58:45.309 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:58:45.309 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmppmk7d5b0:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpiz5l5df0:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:58:45.309 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:58:45.310 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T11:58:45.310 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: if self._apply_service(spec): 2024-01-13T11:58:45.310 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T11:58:45.310 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T11:58:45.310 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T11:58:45.310 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T11:58:45.310 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T11:58:45.310 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T11:58:45.310 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T11:58:45.310 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T11:58:45.310 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T11:58:45.310 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: raise self._value 2024-01-13T11:58:45.310 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T11:58:45.311 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: result = (True, func(*args, **kwds)) 2024-01-13T11:58:45.311 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T11:58:45.311 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: return list(map(*args)) 2024-01-13T11:58:45.311 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:58:45.311 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: return f(*arg) 2024-01-13T11:58:45.311 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:58:45.311 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:58:45.311 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:58:45.311 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: code, '\n'.join(err))) 2024-01-13T11:58:45.311 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:58:45.311 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmppmk7d5b0:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpiz5l5df0:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:58:45.312 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:58:45.312 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:58:45.312 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:58:45.312 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new a27e472f-fdc4-4562-8f46-ffc6a8bda612 2024-01-13T11:58:45.312 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:58:45.312 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:58:45.312 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:58:45.312 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:58:45.312 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:58:45.312 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:58:45.312 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:58:45.313 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:58:45.313 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:58:45.313 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:58:45.313 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:58:45.313 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:58:45.313 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:58:45.313 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:58:45.313 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:58:45.313 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:58:45.313 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:58:45.313 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr 2024-01-13T11:58:45.313 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:58:45.314 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:58:45.314 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:58:45.314 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:58:45.314 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:58:45.314 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:58:45.314 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:58:45.314 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:58:45.314 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:58:45.314 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:58:45.314 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:58:45.314 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:58:45.314 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:58:45.314 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:58:45.315 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:58:45.315 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:58:45.315 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:58:45.315 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:58:45.315 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:58:45.315 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:58:45.315 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:58:45.315 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:58:45.315 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:58:45.315 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:58:45.315 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:58:45.315 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:58:45.316 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:58:45.316 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:58:45.316 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:58:45.316 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:58:45.316 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:58:45.316 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:58:45.316 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:58:45.316 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:58:45.316 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:58:45.316 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:58:45.316 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:58:45.317 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:58:45.317 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: Traceback (most recent call last): 2024-01-13T11:58:45.317 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:58:45.317 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: main() 2024-01-13T11:58:45.317 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:58:45.317 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: r = ctx.func(ctx) 2024-01-13T11:58:45.317 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:58:45.317 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:58:45.317 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:58:45.317 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:58:45.317 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:58:45.317 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:58:45.318 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:58:45.318 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: return func(ctx) 2024-01-13T11:58:45.318 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:58:45.318 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:58:45.318 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:58:45.318 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:58:45.318 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmppmk7d5b0:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpiz5l5df0:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:58:45.318 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: cluster 2024-01-13T11:58:43.215743+0000 mgr.smithi088.xwlxdh (mgr.14182) 394 : cluster [DBG] pgmap v367: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:45.318 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: cluster 2024-01-13T11:58:44.023332+0000 mon.smithi088 (mon.0) 631 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T11:58:45.318 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:45 smithi088 bash[14069]: audit 2024-01-13T11:58:44.471196+0000 mon.smithi088 (mon.0) 632 : audit [DBG] from='client.? 172.21.15.88:0/1002459304' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:58:45.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: cephadm 2024-01-13T11:58:43.211412+0000 mgr.smithi088.xwlxdh (mgr.14182) 392 : cephadm [ERR] executing create_from_spec_one(([('smithi088', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi138', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T11:58:45.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:58:45.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:58:45.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: return f(*arg) 2024-01-13T11:58:45.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:58:45.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:58:45.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:58:45.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: code, '\n'.join(err))) 2024-01-13T11:58:45.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi088/config 2024-01-13T11:58:45.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpzpci51f4:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpu11gzpgj:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:58:45.373 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:58:45.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:58:45.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:58:45.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 43f58b53-2f7a-4c38-8699-eef48a293ad3 2024-01-13T11:58:45.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:58:45.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:58:45.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:58:45.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:58:45.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:58:45.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:58:45.374 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:58:45.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:58:45.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:58:45.375 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:58:45.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:58:45.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:58:45.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:58:45.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:58:45.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:58:45.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:58:45.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:58:45.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:58:45.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:58:45.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:58:45.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:58:45.376 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:58:45.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:58:45.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:58:45.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:58:45.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:58:45.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:58:45.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:58:45.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:58:45.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:58:45.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:58:45.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:58:45.377 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:58:45.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:58:45.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:58:45.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:58:45.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:58:45.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:58:45.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:58:45.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:58:45.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:58:45.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:58:45.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:58:45.378 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:58:45.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:58:45.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:58:45.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:58:45.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:58:45.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:58:45.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:58:45.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:58:45.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:58:45.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:58:45.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:58:45.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:58:45.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:58:45.379 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:58:45.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:58:45.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: main() 2024-01-13T11:58:45.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:58:45.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:58:45.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:58:45.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:58:45.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:58:45.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:58:45.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:58:45.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:58:45.380 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:58:45.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:58:45.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:58:45.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:58:45.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:58:45.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:58:45.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi088 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpzpci51f4:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpu11gzpgj:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:58:45.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: cephadm 2024-01-13T11:58:43.214979+0000 mgr.smithi088.xwlxdh (mgr.14182) 393 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T11:58:45.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: service_id: all-available-devices 2024-01-13T11:58:45.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: service_name: osd.all-available-devices 2024-01-13T11:58:45.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: placement: 2024-01-13T11:58:45.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: host_pattern: '*' 2024-01-13T11:58:45.381 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: spec: 2024-01-13T11:58:45.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: data_devices: 2024-01-13T11:58:45.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: all: true 2024-01-13T11:58:45.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: filter_logic: AND 2024-01-13T11:58:45.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: objectstore: bluestore 2024-01-13T11:58:45.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:58:45.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmppmk7d5b0:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpiz5l5df0:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:58:45.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:58:45.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:58:45.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:58:45.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new a27e472f-fdc4-4562-8f46-ffc6a8bda612 2024-01-13T11:58:45.382 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:58:45.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:58:45.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:58:45.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:58:45.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:58:45.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:58:45.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:58:45.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:58:45.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:58:45.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:58:45.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:58:45.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:58:45.383 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:58:45.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:58:45.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:58:45.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:58:45.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:58:45.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:58:45.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:58:45.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:58:45.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:58:45.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:58:45.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:58:45.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:58:45.384 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:58:45.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:58:45.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:58:45.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:58:45.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:58:45.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:58:45.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:58:45.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:58:45.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:58:45.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:58:45.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:58:45.385 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:58:45.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:58:45.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:58:45.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:58:45.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:58:45.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:58:45.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:58:45.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:58:45.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:58:45.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:58:45.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:58:45.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:58:45.386 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:58:45.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:58:45.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:58:45.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:58:45.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:58:45.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:58:45.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:58:45.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:58:45.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:58:45.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:58:45.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:58:45.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: main() 2024-01-13T11:58:45.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:58:45.387 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:58:45.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:58:45.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:58:45.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:58:45.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:58:45.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:58:45.388 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:58:45.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:58:45.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:58:45.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:58:45.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:58:45.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:58:45.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:58:45.389 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmppmk7d5b0:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpiz5l5df0:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:58:45.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:58:45.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T11:58:45.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: if self._apply_service(spec): 2024-01-13T11:58:45.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T11:58:45.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T11:58:45.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T11:58:45.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T11:58:45.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T11:58:45.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T11:58:45.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T11:58:45.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T11:58:45.390 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T11:58:45.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: raise self._value 2024-01-13T11:58:45.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T11:58:45.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: result = (True, func(*args, **kwds)) 2024-01-13T11:58:45.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T11:58:45.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: return list(map(*args)) 2024-01-13T11:58:45.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T11:58:45.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: return f(*arg) 2024-01-13T11:58:45.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T11:58:45.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T11:58:45.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T11:58:45.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: code, '\n'.join(err))) 2024-01-13T11:58:45.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/mon.smithi138/config 2024-01-13T11:58:45.391 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmppmk7d5b0:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpiz5l5df0:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:58:45.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T11:58:45.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr --> relative data size: 1.0 2024-01-13T11:58:45.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T11:58:45.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new a27e472f-fdc4-4562-8f46-ffc6a8bda612 2024-01-13T11:58:45.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T11:58:45.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T11:58:45.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr stderr: purged osd.0 2024-01-13T11:58:45.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:58:45.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T11:58:45.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr self.prepare() 2024-01-13T11:58:45.392 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:58:45.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:58:45.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T11:58:45.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T11:58:45.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T11:58:45.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T11:58:45.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T11:58:45.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr partitions = get_partitions() 2024-01-13T11:58:45.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr NameError: name 'get_partitions' is not defined 2024-01-13T11:58:45.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:58:45.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr During handling of the above exception, another exception occurred: 2024-01-13T11:58:45.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr 2024-01-13T11:58:45.393 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr Traceback (most recent call last): 2024-01-13T11:58:45.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T11:58:45.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T11:58:45.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T11:58:45.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr self.main(self.argv) 2024-01-13T11:58:45.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T11:58:45.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr return f(*a, **kw) 2024-01-13T11:58:45.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T11:58:45.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T11:58:45.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:58:45.394 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:58:45.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T11:58:45.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T11:58:45.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T11:58:45.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr instance.main() 2024-01-13T11:58:45.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:58:45.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:58:45.395 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T11:58:45.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr self._execute(plan) 2024-01-13T11:58:45.396 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T11:58:45.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) 2024-01-13T11:58:45.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:58:45.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:58:45.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T11:58:45.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr prepare_step.safe_prepare(args) 2024-01-13T11:58:45.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T11:58:45.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T11:58:45.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T11:58:45.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T11:58:45.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T11:58:45.397 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr self.zap_osd() 2024-01-13T11:58:45.398 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T11:58:45.398 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr return func(*a, **kw) 2024-01-13T11:58:45.398 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T11:58:45.398 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T11:58:45.398 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T11:58:45.398 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr '%s' % osd_id or osd_fsid) 2024-01-13T11:58:45.398 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: /usr/bin/docker: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T11:58:45.398 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: Traceback (most recent call last): 2024-01-13T11:58:45.398 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T11:58:45.398 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: main() 2024-01-13T11:58:45.398 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T11:58:45.398 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: r = ctx.func(ctx) 2024-01-13T11:58:45.398 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T11:58:45.399 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:58:45.399 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T11:58:45.399 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:58:45.399 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T11:58:45.399 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:58:45.399 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T11:58:45.399 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: return func(ctx) 2024-01-13T11:58:45.399 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T11:58:45.399 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T11:58:45.399 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: File "/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T11:58:45.399 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T11:58:45.399 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi138 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmppmk7d5b0:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpiz5l5df0:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T11:58:45.400 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: cluster 2024-01-13T11:58:43.215743+0000 mgr.smithi088.xwlxdh (mgr.14182) 394 : cluster [DBG] pgmap v367: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:45.400 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: cluster 2024-01-13T11:58:44.023332+0000 mon.smithi088 (mon.0) 631 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T11:58:45.400 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:45 smithi138 bash[17620]: audit 2024-01-13T11:58:44.471196+0000 mon.smithi088 (mon.0) 632 : audit [DBG] from='client.? 172.21.15.88:0/1002459304' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:58:46.295 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:58:46.309 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:46 smithi088 bash[14069]: cluster 2024-01-13T11:58:45.216721+0000 mgr.smithi088.xwlxdh (mgr.14182) 395 : cluster [DBG] pgmap v368: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:46.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:46 smithi138 bash[17620]: cluster 2024-01-13T11:58:45.216721+0000 mgr.smithi088.xwlxdh (mgr.14182) 395 : cluster [DBG] pgmap v368: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:49.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:49 smithi138 bash[17620]: cluster 2024-01-13T11:58:47.217739+0000 mgr.smithi088.xwlxdh (mgr.14182) 396 : cluster [DBG] pgmap v369: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:49.375 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:58:49.389 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:49 smithi088 bash[14069]: cluster 2024-01-13T11:58:47.217739+0000 mgr.smithi088.xwlxdh (mgr.14182) 396 : cluster [DBG] pgmap v369: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:50.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:50 smithi088 bash[14069]: audit 2024-01-13T11:58:49.387131+0000 mon.smithi088 (mon.0) 633 : audit [DBG] from='client.? 172.21.15.88:0/1747781566' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:58:50.257 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":45,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:58:50.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:50 smithi138 bash[17620]: audit 2024-01-13T11:58:49.387131+0000 mon.smithi088 (mon.0) 633 : audit [DBG] from='client.? 172.21.15.88:0/1747781566' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:58:51.259 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:58:51.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:51 smithi138 bash[17620]: cluster 2024-01-13T11:58:49.218373+0000 mgr.smithi088.xwlxdh (mgr.14182) 397 : cluster [DBG] pgmap v370: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:51.389 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:51 smithi088 bash[14069]: cluster 2024-01-13T11:58:49.218373+0000 mgr.smithi088.xwlxdh (mgr.14182) 397 : cluster [DBG] pgmap v370: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:52.306 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:52 smithi088 bash[14069]: cluster 2024-01-13T11:58:51.219379+0000 mgr.smithi088.xwlxdh (mgr.14182) 398 : cluster [DBG] pgmap v371: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:52.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:52 smithi138 bash[17620]: cluster 2024-01-13T11:58:51.219379+0000 mgr.smithi088.xwlxdh (mgr.14182) 398 : cluster [DBG] pgmap v371: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:54.552 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:58:55.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:55 smithi088 bash[14069]: cluster 2024-01-13T11:58:53.220384+0000 mgr.smithi088.xwlxdh (mgr.14182) 399 : cluster [DBG] pgmap v372: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:55.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:55 smithi088 bash[14069]: audit 2024-01-13T11:58:54.564081+0000 mon.smithi088 (mon.0) 634 : audit [DBG] from='client.? 172.21.15.88:0/4146950091' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:58:55.344 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":45,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:58:55.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:55 smithi138 bash[17620]: cluster 2024-01-13T11:58:53.220384+0000 mgr.smithi088.xwlxdh (mgr.14182) 399 : cluster [DBG] pgmap v372: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:55.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:55 smithi138 bash[17620]: audit 2024-01-13T11:58:54.564081+0000 mon.smithi088 (mon.0) 634 : audit [DBG] from='client.? 172.21.15.88:0/4146950091' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:58:56.345 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:58:56.357 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:56 smithi088 bash[14069]: cluster 2024-01-13T11:58:55.221464+0000 mgr.smithi088.xwlxdh (mgr.14182) 400 : cluster [DBG] pgmap v373: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:56.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:56 smithi138 bash[17620]: cluster 2024-01-13T11:58:55.221464+0000 mgr.smithi088.xwlxdh (mgr.14182) 400 : cluster [DBG] pgmap v373: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:59.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:58:59 smithi138 bash[17620]: cluster 2024-01-13T11:58:57.221969+0000 mgr.smithi088.xwlxdh (mgr.14182) 401 : cluster [DBG] pgmap v374: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:59.390 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:58:59 smithi088 bash[14069]: cluster 2024-01-13T11:58:57.221969+0000 mgr.smithi088.xwlxdh (mgr.14182) 401 : cluster [DBG] pgmap v374: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:58:59.481 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:59:00.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:59:00 smithi088 bash[14069]: audit 2024-01-13T11:58:59.493240+0000 mon.smithi088 (mon.0) 635 : audit [DBG] from='client.? 172.21.15.88:0/1676756416' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:59:00.341 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":45,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:59:00.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:59:00 smithi138 bash[17620]: audit 2024-01-13T11:58:59.493240+0000 mon.smithi088 (mon.0) 635 : audit [DBG] from='client.? 172.21.15.88:0/1676756416' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:59:01.342 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 236db248-b209-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T11:59:01.355 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:59:01 smithi088 bash[14069]: cluster 2024-01-13T11:58:59.222960+0000 mgr.smithi088.xwlxdh (mgr.14182) 402 : cluster [DBG] pgmap v375: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:59:01.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:59:01 smithi138 bash[17620]: cluster 2024-01-13T11:58:59.222960+0000 mgr.smithi088.xwlxdh (mgr.14182) 402 : cluster [DBG] pgmap v375: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:59:02.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:59:02 smithi138 bash[17620]: cluster 2024-01-13T11:59:01.224072+0000 mgr.smithi088.xwlxdh (mgr.14182) 403 : cluster [DBG] pgmap v376: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:59:02.389 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:59:02 smithi088 bash[14069]: cluster 2024-01-13T11:59:01.224072+0000 mgr.smithi088.xwlxdh (mgr.14182) 403 : cluster [DBG] pgmap v376: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:59:04.616 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-01-13T11:59:05.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:59:05 smithi088 bash[14069]: cluster 2024-01-13T11:59:03.225141+0000 mgr.smithi088.xwlxdh (mgr.14182) 404 : cluster [DBG] pgmap v377: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:59:05.140 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:59:05 smithi088 bash[14069]: audit 2024-01-13T11:59:04.627982+0000 mon.smithi088 (mon.0) 636 : audit [DBG] from='client.? 172.21.15.88:0/1819398927' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:59:05.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:59:05 smithi138 bash[17620]: cluster 2024-01-13T11:59:03.225141+0000 mgr.smithi088.xwlxdh (mgr.14182) 404 : cluster [DBG] pgmap v377: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T11:59:05.372 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:59:05 smithi138 bash[17620]: audit 2024-01-13T11:59:04.627982+0000 mon.smithi088 (mon.0) 636 : audit [DBG] from='client.? 172.21.15.88:0/1819398927' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T11:59:05.472 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":45,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705146532,"num_remapped_pgs":0} 2024-01-13T11:59:05.473 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 779, in ceph_osds while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (121) after waiting for 120 seconds 2024-01-13T11:59:05.474 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-13T11:59:05.531 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-13T11:59:05.547 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-01-13T11:59:05.547 DEBUG:teuthology.orchestra.run.smithi088:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-13T11:59:05.582 DEBUG:teuthology.orchestra.run.smithi138:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-13T11:59:05.596 INFO:tasks.cephadm:Stopping all daemons... 2024-01-13T11:59:05.596 INFO:tasks.cephadm.mon.smithi088:Stopping mon.smithi088... 2024-01-13T11:59:05.596 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-236db248-b209-11ee-95ac-87774f69a715@mon.smithi088 2024-01-13T11:59:05.818 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:59:05 smithi088 systemd[1]: Stopping Ceph mon.smithi088 for 236db248-b209-11ee-95ac-87774f69a715... 2024-01-13T11:59:05.818 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:59:05 smithi088 bash[52590]: Error response from daemon: No such container: ceph-236db248-b209-11ee-95ac-87774f69a715-mon.smithi088 2024-01-13T11:59:05.818 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:59:05 smithi088 bash[14069]: debug 2024-01-13T11:59:05.828+0000 7fc58b826700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.smithi088 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-13T11:59:06.075 INFO:journalctl@ceph.mon.smithi088.smithi088.stdout:Jan 13 11:59:05 smithi088 bash[14069]: debug 2024-01-13T11:59:05.828+0000 7fc58b826700 -1 mon.smithi088@0(leader) e2 *** Got Signal Terminated *** 2024-01-13T11:59:06.923 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-236db248-b209-11ee-95ac-87774f69a715@mon.smithi088.service' 2024-01-13T11:59:06.966 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-13T11:59:06.966 INFO:tasks.cephadm.mon.smithi088:Stopped mon.smithi088 2024-01-13T11:59:06.966 INFO:tasks.cephadm.mon.smithi138:Stopping mon.smithi138... 2024-01-13T11:59:06.966 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ceph-236db248-b209-11ee-95ac-87774f69a715@mon.smithi138 2024-01-13T11:59:07.295 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:59:07 smithi138 systemd[1]: Stopping Ceph mon.smithi138 for 236db248-b209-11ee-95ac-87774f69a715... 2024-01-13T11:59:07.295 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:59:07 smithi138 bash[23195]: Error response from daemon: No such container: ceph-236db248-b209-11ee-95ac-87774f69a715-mon.smithi138 2024-01-13T11:59:07.295 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:59:07 smithi138 bash[17620]: debug 2024-01-13T11:59:07.148+0000 7f3d02152700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.smithi138 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-13T11:59:07.295 INFO:journalctl@ceph.mon.smithi138.smithi138.stdout:Jan 13 11:59:07 smithi138 bash[17620]: debug 2024-01-13T11:59:07.148+0000 7f3d02152700 -1 mon.smithi138@1(peon) e2 *** Got Signal Terminated *** 2024-01-13T11:59:07.994 DEBUG:teuthology.orchestra.run.smithi138:> sudo pkill -f 'journalctl -f -n 0 -u ceph-236db248-b209-11ee-95ac-87774f69a715@mon.smithi138.service' 2024-01-13T11:59:08.044 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-13T11:59:08.044 INFO:tasks.cephadm.mon.smithi138:Stopped mon.smithi138 2024-01-13T11:59:08.044 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 236db248-b209-11ee-95ac-87774f69a715 --force --keep-logs 2024-01-13T11:59:19.464 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 236db248-b209-11ee-95ac-87774f69a715 --force --keep-logs 2024-01-13T11:59:29.117 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-13T11:59:29.133 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-13T11:59:29.148 INFO:tasks.cephadm:Archiving crash dumps... 2024-01-13T11:59:29.149 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7516049/remote/ubuntu@smithi088.front.sepia.ceph.com/crash 2024-01-13T11:59:29.150 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash -- . 2024-01-13T11:59:29.189 INFO:teuthology.orchestra.run.smithi088.stderr:tar: /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash: Cannot open: No such file or directory 2024-01-13T11:59:29.189 INFO:teuthology.orchestra.run.smithi088.stderr:tar: Error is not recoverable: exiting now 2024-01-13T11:59:29.191 DEBUG:teuthology.misc:Transferring archived files from smithi138:/var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7516049/remote/ubuntu@smithi138.front.sepia.ceph.com/crash 2024-01-13T11:59:29.192 DEBUG:teuthology.orchestra.run.smithi138:> sudo tar c -f - -C /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash -- . 2024-01-13T11:59:29.208 INFO:teuthology.orchestra.run.smithi138.stderr:tar: /var/lib/ceph/236db248-b209-11ee-95ac-87774f69a715/crash: Cannot open: No such file or directory 2024-01-13T11:59:29.208 INFO:teuthology.orchestra.run.smithi138.stderr:tar: Error is not recoverable: exiting now 2024-01-13T11:59:29.209 INFO:tasks.cephadm:Checking cluster log for badness... 2024-01-13T11:59:29.209 DEBUG:teuthology.orchestra.run.smithi088:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-01-13T11:59:29.246 INFO:teuthology.orchestra.run.smithi088.stderr:grep: /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715/ceph.log: No such file or directory 2024-01-13T11:59:29.248 INFO:tasks.cephadm:Compressing logs... 2024-01-13T11:59:29.248 DEBUG:teuthology.orchestra.run.smithi088:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-13T11:59:29.294 DEBUG:teuthology.orchestra.run.smithi138:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-13T11:59:29.307 INFO:teuthology.orchestra.run.smithi088.stderr:find: '/var/log/rbd-target-api': No such file or directory 2024-01-13T11:59:29.309 INFO:teuthology.orchestra.run.smithi138.stderr:find: '/var/log/rbd-target-api': No such file or directory 2024-01-13T11:59:29.314 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715/ceph-mgr.smithi088.xwlxdh.log 2024-01-13T11:59:29.314 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715/ceph-mon.smithi088.log 2024-01-13T11:59:29.315 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715/ceph-volume.log 2024-01-13T11:59:29.315 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/236db248-b209-11ee-95ac-87774f69a715/ceph-mgr.smithi088.xwlxdh.log: /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715/ceph-mon.smithi088.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-01-13T11:59:29.321 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715/ceph-mgr.smithi138.divzmq.log 2024-01-13T11:59:29.321 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715/ceph-mon.smithi138.log 2024-01-13T11:59:29.322 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715/ceph-volume.log /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715/ceph-mgr.smithi138.divzmq.log: 2024-01-13T11:59:29.322 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/236db248-b209-11ee-95ac-87774f69a715/ceph-volume.log: /var/log/ceph/cephadm.log: 93.9% -- replaced with /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715/ceph-volume.log.gz 2024-01-13T11:59:29.323 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/236db248-b209-11ee-95ac-87774f69a715/ceph-mon.smithi138.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-01-13T11:59:29.326 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715/ceph-volume.log: 92.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-13T11:59:29.327 INFO:teuthology.orchestra.run.smithi088.stderr: 92.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-13T11:59:29.328 INFO:teuthology.orchestra.run.smithi138.stderr: 92.3% -- replaced with /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715/ceph-mgr.smithi138.divzmq.log.gz 2024-01-13T11:59:29.335 INFO:teuthology.orchestra.run.smithi138.stderr: 93.9% -- replaced with /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715/ceph-volume.log.gz 2024-01-13T11:59:29.424 INFO:teuthology.orchestra.run.smithi088.stderr: 91.1% -- replaced with /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715/ceph-mgr.smithi088.xwlxdh.log.gz 2024-01-13T11:59:29.432 INFO:teuthology.orchestra.run.smithi138.stderr: 92.8% -- replaced with /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715/ceph-mon.smithi138.log.gz 2024-01-13T11:59:29.433 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-01-13T11:59:29.433 INFO:teuthology.orchestra.run.smithi138.stderr:real 0m0.135s 2024-01-13T11:59:29.433 INFO:teuthology.orchestra.run.smithi138.stderr:user 0m0.132s 2024-01-13T11:59:29.434 INFO:teuthology.orchestra.run.smithi138.stderr:sys 0m0.026s 2024-01-13T11:59:29.624 INFO:teuthology.orchestra.run.smithi088.stderr: 89.8% -- replaced with /var/log/ceph/236db248-b209-11ee-95ac-87774f69a715/ceph-mon.smithi088.log.gz 2024-01-13T11:59:29.650 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-01-13T11:59:29.650 INFO:teuthology.orchestra.run.smithi088.stderr:real 0m0.329s 2024-01-13T11:59:29.650 INFO:teuthology.orchestra.run.smithi088.stderr:user 0m0.440s 2024-01-13T11:59:29.651 INFO:teuthology.orchestra.run.smithi088.stderr:sys 0m0.022s 2024-01-13T11:59:29.651 INFO:tasks.cephadm:Archiving logs... 2024-01-13T11:59:29.651 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7516049/remote/ubuntu@smithi088.front.sepia.ceph.com/log 2024-01-13T11:59:29.652 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-13T11:59:29.811 DEBUG:teuthology.misc:Transferring archived files from smithi138:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7516049/remote/ubuntu@smithi138.front.sepia.ceph.com/log 2024-01-13T11:59:29.812 DEBUG:teuthology.orchestra.run.smithi138:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-13T11:59:29.848 INFO:tasks.cephadm:Removing cluster... 2024-01-13T11:59:29.848 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 236db248-b209-11ee-95ac-87774f69a715 --force 2024-01-13T11:59:31.207 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 236db248-b209-11ee-95ac-87774f69a715 --force 2024-01-13T11:59:32.588 INFO:tasks.cephadm:Removing cephadm ... 2024-01-13T11:59:32.588 DEBUG:teuthology.orchestra.run.smithi088:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-13T11:59:32.596 DEBUG:teuthology.orchestra.run.smithi138:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-13T11:59:32.603 INFO:tasks.cephadm:Teardown complete 2024-01-13T11:59:32.603 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 779, in ceph_osds while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (121) after waiting for 120 seconds 2024-01-13T11:59:32.820 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=68bc6f9a6d2f4e368269cd4f679ef700 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 779, in ceph_osds while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (121) after waiting for 120 seconds 2024-01-13T11:59:32.823 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-01-13T11:59:32.831 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2024-01-13T11:59:32.839 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi088:/dev/vg_nvme/lv_1... 2024-01-13T11:59:32.839 DEBUG:teuthology.orchestra.run.smithi088:> sudo nvme disconnect -n lv_1 2024-01-13T11:59:33.129 INFO:teuthology.orchestra.run.smithi088.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-01-13T11:59:33.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T11:59:33.130 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi088:/dev/vg_nvme/lv_2... 2024-01-13T11:59:33.131 DEBUG:teuthology.orchestra.run.smithi088:> sudo nvme disconnect -n lv_2 2024-01-13T11:59:33.408 INFO:teuthology.orchestra.run.smithi088.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-01-13T11:59:33.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T11:59:33.410 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi088:/dev/vg_nvme/lv_3... 2024-01-13T11:59:33.411 DEBUG:teuthology.orchestra.run.smithi088:> sudo nvme disconnect -n lv_3 2024-01-13T11:59:33.676 INFO:teuthology.orchestra.run.smithi088.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-01-13T11:59:33.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T11:59:33.678 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi088:/dev/vg_nvme/lv_4... 2024-01-13T11:59:33.678 DEBUG:teuthology.orchestra.run.smithi088:> sudo nvme disconnect -n lv_4 2024-01-13T11:59:33.936 INFO:teuthology.orchestra.run.smithi088.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-01-13T11:59:33.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T11:59:33.938 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-01-13T11:59:33.939 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/scratch_devs 2024-01-13T11:59:33.953 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi138:/dev/vg_nvme/lv_1... 2024-01-13T11:59:33.954 DEBUG:teuthology.orchestra.run.smithi138:> sudo nvme disconnect -n lv_1 2024-01-13T11:59:34.243 INFO:teuthology.orchestra.run.smithi138.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-01-13T11:59:34.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T11:59:34.245 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi138:/dev/vg_nvme/lv_2... 2024-01-13T11:59:34.245 DEBUG:teuthology.orchestra.run.smithi138:> sudo nvme disconnect -n lv_2 2024-01-13T11:59:34.555 INFO:teuthology.orchestra.run.smithi138.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-01-13T11:59:34.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T11:59:34.557 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi138:/dev/vg_nvme/lv_3... 2024-01-13T11:59:34.557 DEBUG:teuthology.orchestra.run.smithi138:> sudo nvme disconnect -n lv_3 2024-01-13T11:59:34.827 INFO:teuthology.orchestra.run.smithi138.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-01-13T11:59:34.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T11:59:34.828 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi138:/dev/vg_nvme/lv_4... 2024-01-13T11:59:34.829 DEBUG:teuthology.orchestra.run.smithi138:> sudo nvme disconnect -n lv_4 2024-01-13T11:59:35.083 INFO:teuthology.orchestra.run.smithi138.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-01-13T11:59:35.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T11:59:35.085 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-01-13T11:59:35.085 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/scratch_devs 2024-01-13T11:59:35.099 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-13T11:59:35.111 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-13T11:59:35.111 DEBUG:teuthology.orchestra.run.smithi088:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-13T11:59:35.113 DEBUG:teuthology.orchestra.run.smithi138:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-13T11:59:35.130 INFO:teuthology.orchestra.run.smithi088.stdout: remote refid st t when poll reach delay offset jitter 2024-01-13T11:59:35.131 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================== 2024-01-13T11:59:35.131 INFO:teuthology.orchestra.run.smithi088.stdout:*hv01.front.sepi 44.4.53.2 2 u 42 64 377 0.123 1.499 0.404 2024-01-13T11:59:35.131 INFO:teuthology.orchestra.run.smithi088.stdout:+hv02.front.sepi 63.231.80.2 3 u 46 64 377 0.113 1.467 0.434 2024-01-13T11:59:35.131 INFO:teuthology.orchestra.run.smithi088.stdout:+hv03.front.sepi 74.6.168.72 3 u 45 64 377 0.092 -0.330 0.716 2024-01-13T11:59:35.131 INFO:teuthology.orchestra.run.smithi088.stdout: hv04.front.sepi .INIT. 16 u - 256 0 0.000 0.000 0.000 2024-01-13T11:59:35.162 INFO:teuthology.orchestra.run.smithi138.stdout: remote refid st t when poll reach delay offset jitter 2024-01-13T11:59:35.163 INFO:teuthology.orchestra.run.smithi138.stdout:============================================================================== 2024-01-13T11:59:35.163 INFO:teuthology.orchestra.run.smithi138.stdout:*hv01.front.sepi 44.4.53.2 2 u 36 64 377 0.085 1.286 0.119 2024-01-13T11:59:35.163 INFO:teuthology.orchestra.run.smithi138.stdout:+hv02.front.sepi 63.231.80.2 3 u 42 64 377 0.098 0.683 0.109 2024-01-13T11:59:35.163 INFO:teuthology.orchestra.run.smithi138.stdout:+hv03.front.sepi 74.6.168.72 3 u 47 64 377 0.088 -0.263 0.181 2024-01-13T11:59:35.163 INFO:teuthology.orchestra.run.smithi138.stdout: hv04.front.sepi .INIT. 16 u - 256 0 0.000 0.000 0.000 2024-01-13T11:59:35.164 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-13T11:59:35.173 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-13T11:59:35.173 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-13T11:59:35.182 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-13T11:59:35.189 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-13T11:59:35.197 INFO:teuthology.task.internal:Duration was 1227.023965 seconds 2024-01-13T11:59:35.198 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-13T11:59:35.207 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-13T11:59:35.207 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-13T11:59:35.209 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-13T11:59:35.255 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-13T11:59:35.255 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi088.front.sepia.ceph.com 2024-01-13T11:59:35.256 DEBUG:teuthology.orchestra.run.smithi088:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-13T11:59:35.310 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi138.front.sepia.ceph.com 2024-01-13T11:59:35.311 DEBUG:teuthology.orchestra.run.smithi138:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-13T11:59:35.326 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-13T11:59:35.326 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-13T11:59:35.354 DEBUG:teuthology.orchestra.run.smithi138:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-13T11:59:35.422 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-13T11:59:35.422 DEBUG:teuthology.orchestra.run.smithi088:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-13T11:59:35.424 DEBUG:teuthology.orchestra.run.smithi138:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-13T11:59:35.589 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-13T11:59:35.598 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-13T11:59:35.599 DEBUG:teuthology.orchestra.run.smithi088:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-13T11:59:35.615 DEBUG:teuthology.orchestra.run.smithi138:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-13T11:59:35.631 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-13T11:59:35.665 DEBUG:teuthology.orchestra.run.smithi088:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-13T11:59:35.667 DEBUG:teuthology.orchestra.run.smithi138:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-13T11:59:35.679 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = core 2024-01-13T11:59:35.687 INFO:teuthology.orchestra.run.smithi138.stdout:kernel.core_pattern = core 2024-01-13T11:59:35.718 DEBUG:teuthology.orchestra.run.smithi088:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-13T11:59:35.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T11:59:35.754 DEBUG:teuthology.orchestra.run.smithi138:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-13T11:59:35.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T11:59:35.767 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-13T11:59:35.777 INFO:teuthology.task.internal:Transferring archived files... 2024-01-13T11:59:35.777 DEBUG:teuthology.misc:Transferring archived files from smithi088:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7516049/remote/smithi088 2024-01-13T11:59:35.778 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-13T11:59:35.829 DEBUG:teuthology.misc:Transferring archived files from smithi138:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7516049/remote/smithi138 2024-01-13T11:59:35.830 DEBUG:teuthology.orchestra.run.smithi138:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-13T11:59:35.861 INFO:teuthology.task.internal:Removing archive directory... 2024-01-13T11:59:35.861 DEBUG:teuthology.orchestra.run.smithi088:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-13T11:59:35.865 DEBUG:teuthology.orchestra.run.smithi138:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-13T11:59:35.904 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-13T11:59:35.915 INFO:teuthology.task.internal:Not uploading archives. 2024-01-13T11:59:35.916 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-13T11:59:35.926 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-13T11:59:35.926 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-13T11:59:35.928 DEBUG:teuthology.orchestra.run.smithi138:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-13T11:59:35.934 INFO:teuthology.orchestra.run.smithi088.stdout: 1048617 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 13 11:59 /home/ubuntu/cephtest 2024-01-13T11:59:35.953 INFO:teuthology.orchestra.run.smithi138.stdout: 1048620 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 13 11:59 /home/ubuntu/cephtest 2024-01-13T11:59:35.956 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-13T11:59:35.966 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-13T11:59:35.987 INFO:teuthology.nuke:Checking targets against current locks 2024-01-13T11:59:36.015 DEBUG:teuthology.nuke:shortname: smithi088 2024-01-13T11:59:36.015 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-13T11:59:36.043 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7516049', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-13 11:29:03.519793', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2024-01-13T11:59:36.046 DEBUG:teuthology.nuke:shortname: smithi138 2024-01-13T11:59:36.047 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-13T11:59:36.068 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi138.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7516049', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-13 11:29:03.518248', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2024-01-13T11:59:36.095 INFO:teuthology.orchestra.console.smithi088:Power off 2024-01-13T11:59:36.096 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-13T11:59:36.121 DEBUG:teuthology.orchestra.console.smithi088:power off output: Chassis Power Control: Down/Off 2024-01-13T11:59:36.121 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-13T11:59:36.138 INFO:teuthology.orchestra.console.smithi138:Power off 2024-01-13T11:59:36.138 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-13T11:59:36.148 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on 2024-01-13T11:59:36.161 DEBUG:teuthology.orchestra.console.smithi138:power off output: Chassis Power Control: Down/Off 2024-01-13T11:59:36.162 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-13T11:59:36.189 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is on 2024-01-13T11:59:40.149 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-13T11:59:40.190 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-13T11:59:40.258 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on 2024-01-13T11:59:40.301 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is on 2024-01-13T11:59:44.260 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-13T11:59:44.302 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-13T11:59:44.371 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on 2024-01-13T11:59:44.412 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is on 2024-01-13T11:59:48.376 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-13T11:59:48.414 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-13T11:59:48.486 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is off 2024-01-13T11:59:48.524 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is off 2024-01-13T11:59:48.587 INFO:teuthology.orchestra.console.smithi088:Power off completed 2024-01-13T11:59:48.625 INFO:teuthology.orchestra.console.smithi138:Power off completed 2024-01-13T11:59:48.729 INFO:teuthology.run:Summary data: description: rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} duration: 1227.023964881897 failure_reason: reached maximum tries (121) after waiting for 120 seconds owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=68bc6f9a6d2f4e368269cd4f679ef700 status: fail success: false 2024-01-13T11:59:48.730 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-13T11:59:48.813 INFO:teuthology.run:FAIL